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THE 

SINGLE-BOARD 
COMPUTERS at 
ISSUE st 


Technology, former Linux 
Marketing Guy (tm), former 
openSUSE Board Member...and 
current Deputy Editor of Linux 


1980s computer. Journal as well as host of the 
(aptly named) Lunduke Show. 


When | was a child in the 1980s, | had a computer—a very 


It had a hefty, rectangular, grey case made of some sort of 
industrial sheet metal. Two plain (but rather large), square 
buttons adorned the front, begging to be pressed: “Reset” 
and “Turbo”. On the right side of the case, far in the back 
(nearly out of reach), sat an almost comically large, red 
power switch. It was the kind of lever that would look right 
at home in an action movie—used to cut the electricity to all 
of New York City. 


When you “threw the switch”, the PC turned on witha 
deeply satisfying, soul-reverberating, “ka-THUNK”, 


Inside, sat an Intel 286 CPU decked out with 640k of RAM, 
which, as some unnamed person may or may not have said, 
“ought to be enough for anybody”. For mass storage, it 

had a big, double tall hard drive. The connection for this 
drive wasn’t SATA, or SCSI, or even IDE. We’re talking about 
an MFM connection here, baby (MFM stands for Modified 
Frequency Modulation). As a child, | simply assumed MFM 
had something to do with the fact that you could hear the 
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hard drive spinning up from down the street. 


| kid, | kid. You couldn’t actually hear the hard drive—not over the roar of the fan in 
the power supply. 


It was, to say the least, a beast—beastly in size, beastly in power usage and 
beastly in price. 


Flash-forward [counts on fingers, gets depressed at own age, downs a pint of 
ice cream, resumes writing article ] 35 years later. We now have single-board 
computers (SBCs) with no fans—heck, no moving parts whatsoever—running 


completely silently. 


These SBCs have several hundred times (in some cases, several thousand times) 
the RAM. Ditto for storage. With significantly faster networking (including 
wireless, which wasn’t even a thing on that old 286) and processing speed that, 
even among the slowest SBCs, is so much faster, it’s almost mind-boggling. 


All of this is contained within a physical size often smaller than a credit card and at a 
price somewhere roughly between one hamburger and...a couple more hamburgers. 


These small, silent, low-power, low-cost computers have changed things. They’ve 
made general-purpose computing more affordable (and durable), bringing down 
costs in data centers and allowing solo makers and small companies to create 
computer-driven hardware projects that would have been nearly impossible to 
tackle in days gone by. 


Here in 2019, we’ve even got a whole heaping helping of SBCs from which to choose: 
Arduino, BeagleBoard, Gumstix, ODROID, Pine64, Raspberry Pi—the list goes on and 
on. We are spoiled for choices. 


n this issue of Linux Journal, we look at SBCs from multiple angles. We start the 
Deep Dive section with “Arduino from the Command Line: Break Free from the GUI 
with Git and Vim!”—a detailed look into how to utilize the newly created arduino-cli 
application to work with Arduino devices directly from, you guessed it, the 
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command line, written by Matthew Hoskins, the Senior Enterprise Architect at New 
Jersey Institute of Technology. 


Then, in “Indie Makers Using Single-Board Computers”, | sit down with the minds 
behind two solo projects based around the Raspberry Pi Zero (the smallest of the 
Raspberry Pi family): the small, PDA-like NoodlePi and the “oh my gosh, it’s like an 
itty-bitty gameboy” TinyPi. 


Next up, Professor Jan Newmarch takes us ona guided tour of how to set up Mycroft 
(a personal digital assistant that is open-source and Linux-based) ona Raspberry Pi 
3—complete with a look at how to build Mycroft “skills” (in this case, interfacing with 
smart bulbs) using Python. 


Turning to the Enterprise space—because these single-board computers aren’t just for 
hobbyist projects—Charles Fisher gives us a review of the newly released Raspberry Pi 
3 version of Oracle Linux, including instructions for installing Oracle on the Pi 3 using 

the Btrfs filesystem. 


We finish with a look into PiBox—a custom Linux distribution for the Raspberry Pi, 
with cross-compiled applications, intended for distributed media playback, written by 
the man behind the project himself, Michael J. Hammel. 

These single-board computers are incredibly versatile. The size and price point alone 
open up whole worlds of possibilities that seemed out of reach just a few years ago, 
both in mobile computing and enterprise-grade server farms—truly amazing. 


Though, if l’m being honest, | do rather miss that big, red power switch. 


Also, now | want a hamburger. @ 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email |jeditor@linuxjournal.com. 
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The Digital 
Unconformity 


Will our digital lives leave a fossil record? Or any 
record at alle 


In the library of Earth’s history, there are missing books. All 
were written in rock that is now gone. The greatest example 
of “gone” rock first was observed by in 
1869, on his expedition by boat through the Grand Canyon. 
Floating down the Colorado river, he saw the canyon’s mile- 
thick layers of reddish sedimentary rock resting on a basement 
of gray non-sedimentary rock, and he correctly assumed that 
the upper layers did not continue from the bottom one. He 
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FROM THE EDITOR 


knew time had passed between the basement rock and the floors of rock above it, 
but he didn’t know how much. The answer turned out to be more than a billion years. 
The walls of the Grand Canyon say nothing about what happened during that time. 
Geology calls that nothing an unconformity. 


In fact, Powell’s unconformity prevails worldwide. The name for this worldwide 
missing rock is the Great Unconformity. Because of that unconformity, geology 
knows comparatively little about what happened in the world through stretches of 
time ranging regionally up to 1.6 billion years. All of those stretches end abruptly 
with the Cambrian Explosion, which began about 541 million years ago. Many 
theories attempt to explain what erased all that geological history, but the prevailing 
paradigm is perhaps best expressed in “Neoproterozoic glacial origin of the Great 
Unconformity”, published on the last day of 2018 by nine geologists writing for the 
National Academy of Sciences. 


Put simply, they blame snow. Lots of it—enough to turn the planet into one giant 
snowball, already informally called Snowball Earth. A more accurate name for this 
time would be Glacierball Earth, because glaciers, all formed from snow, apparently 
covered most or all of Earth’s land during the Great Unconformity—and most or all 
of the seas as well. 


The relevant fact about glaciers is that they don’t sit still. They spread and slide 
sideways, pressing and pushing immensities of accumulated ice down on landscapes 
that they pulverize and scrape against adjacent landscapes, abrading their way through 
mountains and across hills and plains like a trowel spreading wet cement. Thus, it 
seems glaciers scraped a vastness of geological history off the Earth’s surface and let 
plate tectonics hide the rest of the evidence. As a result, the stories of Earth’s missing 
history are told only by younger rock that remembers only that a layer of moving ice 
had erased pretty much everything other than a signature on its work. 


| bring all this up because | see something analogous to Glacierball Earth happening 
right now, right here, across our new worldwide digital sphere. A snowstorm of bits 
is falling on the virtual surface of that virtual sphere, which itself is made of bits even 
more provisional and temporary than the glaciers that once covered the physical 
Earth. All of this digital storm, vivid and present in our current moment in time, is not 
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only doomed to vanish, but it lacks even a glacier’s talent for accumulation. 


There is nothing about a bit that lends itself to persistence, other than the media it 


is written on, if it is written at all. Form follows function, and right now, most digital 
functions, even those we call “storage”, are temporary. The largest commercial 
facilities for storing digital goods are what we fittingly call “clouds”. By design, these 
are built to remember no more of what they contained than does an empty closet. 
Stop paying for cloud storage, and away goes your stuff, leaving no fossil imprints. 
Old hard drives, CDs and DVDs might persist in landfills, but people in the far future 
may look at a CD or a DVD the way a geologist today looks at Cambrian zircons: as 
signatures of digital activity in a lost period of time. If those fossils speak of what’s 
happening now at all, it will be of a self-erasing Digital Earth that began in the late 
20th century. 


This isn’t my theory. It comes from my wife, who has long claimed that future 
historians will look on our digital age as an invisible one, because it sucks so royally at 
archiving itself. | think she’s right. 


For example, this laptop currently sits atop a stack of books on my desk. Two of those 
books are self-published compilations of essays | wrote about technology in the mid- 
1980s, mostly for publications that are long gone. The originals are on floppy disks 
that can be read only by PCs and apps of that time, some of which are buried in lower 
strata of boxes in my garage. | just found a floppy with some of those essays. (It’s the 
one with a blue edge in the wood case near the right end of the photo.) But I’'ll need 
to find an old machine to read it. Old apps too, all of them also on floppies. If those 
still retain readable files, | am sure there are ways to recover at least the raw ASCII 
text. But I’m still betting the paper copies of the books under this laptop will live a 

lot longer than the floppies or the stored PCs will—if those aren’t already bricked by 
decades of un-use. 


As for other media, the prospect isn’t any better. 
At the base of my video collection is a stratum of VHS videotapes, atop of which 


are strata of Video8 and Hi8 tapes, and then one of digital stuff burned onto CDs 
and stored in hard drives, most of which have been disconnected for years. Some of 
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those drives have interfaces and connections no longer supported by any computers 
being made today. Although I’ve saved machines to play all of them, none I’ve checked 
still work. One choked to death on a CD | stuck in it. And that was just one failure 
among many that stopped me from making Christmas presents of family memories 
recorded on old tapes and DVDs. | may take up the project again sometime before 
next Christmas, but the odds are long against that (hey, I’m busy) and short toward 
the chance that nobody will ever see or hear those recordings again. And I’m not even 
counting my parents’ 8mm and 16mm movies made from the 1930s to the 1960s. In 
1989, my sister and | had all of those copied over to VHS tape. We then recorded my 
mother annotating the tapes onto companion cassette tapes while we watched the 
show. | still have the original film in a box somewhere, but | can’t find any of the tapes. 


The base stratum of my audio past is a few dozen open reel tapes recorded in the 
1950s and 1960s. Above that are cassette and microcassete tapes, plus some Sony 
MiniDisks recorded in ATRAC, a proprietary and incompatible compression algorithm 
now used by nobody, including Sony. Although | do have ways to play some (but 

not all) of those, I’m cautious about converting any of them to digital formats (Ogg, 
MPEG or whatever), because all digital storage media becomes obsolete, dies, or 
both—as do formats, algorithms and codecs. Already | have dozens of dead external 
hard drives in boxes and drawers. And no commercial cloud service is committed to 


digital preservation in perpetuity in the absence of payment. This means my saved 
files in clouds are sure to be flushed after neither my heirs nor | continue paying for 
their preservation. 


Same goes for my photographs. My old photographs are stored in boxes and 
albums of photos, negatives and Kodak slide carousels. My digital photographs 

are spread across a mess of duplicated back-up drives totaling many terabytes, 

plus a handful of CDs. About 60,000 photos are exposed to the world on Flickr’s 
cloud, where | maintain two Pro accounts (here and here) for $50/year a piece. 
More are in the Berkman Klein Center’s pro account (here) and Linux Journal’s 
(here). It is unclear currently whether any of that will survive after any of those 
entities stop paying the yearly fee. SmugMug, which now owns Flickr, has said 
some encouraging things about photos such as mine, all of which are Creative 
Commons-licensed to encourage re-use. But, as Geoffrey West tells us, companies 


are mortal. All of them die. 
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As for my digital works as a whole (or anybody’s), there is great promise in what the 
Internet Archive and Wikimedia Commons do, but there is no guarantee that either 
will last for decades more, much less for centuries or millennia. And neither are able 

to archive everything that matters (much as they might like to). 


It also should be sobering to recognize that we only rent the “sites”, “locations” and 
“addresses” at “domains” that we talk about “owning”. The plain and awful fact is that 
nobody “owns” a domain on the internet. They pay a sum to a registrar for the right 
to use a domain name for a finite period of time. There are no permanent domain 
names or IP addresses. In the digital world, finitude rules. 


So the historic progression | see, and try to illustrate in the photo at the beginning of 
this article, is from hard physical records through digital ones we hold for ourselves, 
and then up into clouds that go away, inevitably. Everything digital is snow falling and 
disappearing on the waters of time. 


Will there ever be a way to save for the very long term what we ironically call 
our digita 


|.“ 


assets” for more than a few dozen years? Or is all of it doomed by 
its own nature to disappear, leaving little more evidence of its passage than a 
digital unconformity? 


| can’t think of any technical questions more serious than those two. 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email ljeditor@linuxjournal.com. 
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The Light at the End of the Pipe 


| read Kyle Rankin’s “Put Down the Pipe” article from the January 2019 issue with 
interest, as I’m a big fan of Bash and love to learn new tricks. | can’t help feeling that 


the last example using find missed one of the most powerful options to find: 
-exec. This is a built-in xargs replacement and is very powerful. One of my favourite 
uses is this: 


find . -name "x.c" -exec grep "thing" {} /dev/null \; 


| 


This searches all “.c” files beneath the current working directory for “thing” and 
outputs the name of the file and the line containing “thing”. The {} is replaced by 
the files found, one by one. /dev/null forces grep to output the name of the file 
containing thing, as it doesn’t otherwise do so if there’s only one file passed to it. 
The beauty of -exec is that it can be used with any other shell command. The above 
example improves on grep —Ras there are many other options that can be used with 


find to filter the files searched (for example, —type). 
—Steve Kerr 


Thanks for “Have a Plan for Netplan" 

| just wanted to thank Shawn Powers for his article “Have a Plan for Netplan” in 

the October 2018 issue. | also wanted to mention a configuration detail that may be 
related to the peculiar DNS mis-behaviors that Shawn reported in the article. As it 
turns out, similar mis-behaviors also had been a problem for my network configuration. 


My Xubuntu 18.04 desktop system uses a Netplan YAML file with networkd as 

the renderer. | installed Stubby with the intent of using DNS over TLS for the 

name service. However, when | first applied the Netplan configuration, | also 

was observing (uncontrollable) DNS-resolution behaviors similar to those 
reported in the article. After reading the article, | realized that | could disable the 
systemd-resolved service and allow the Stubby resolver to exclusively take over the 
DNS-resolution function. This approach ultimately solved my problems—thanks! 
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Being a Linux-newbie, it has taken considerable time and effort to figure out what ( 
think) was happening. Ill provide a dialog below to describe what | did and what may 
be the cause of the problems that led to the DNS mis-behaviors. 


To start, it is not readily apparent that systemd-resolved provides sufficient controllability 


from Netplan—for preventing automatic access to the gateway’s DNS server. 


For example, configuring with NetworkManager via the GNOME desktop allows 
for invocation of a “DNS Automatic OFF” control. This option is available from the 


NetworkManager configuration GUI, but | couldn’t find a similar directive for Netplan. 
’m thinking that NetworkManager has the necessary “hooks” for configuring the 
systemd-resolved service with “DNS Automatic OFF”, whereas Netplan does not? 


There’s a possibly related Netplan bug report here. 


| used this web page for installing Stubby on Ubuntu 18.04, which uses 
NetworkManager and sets “DNS Automatic OFF” via the GNOME desktop. 


| performed all installation steps up to the section titled “Switching to Stubby”, which 
describes configuration using NetworkManager in that section. 


Of course, for my implementation, the reference in the YAML file, setting the 
renderer to “NetworkManager” was removed and replaced with networkd. 


| found that using netplan and running stubby as the replacement stub resolver can 
be made to work if systemd-resolved is disabled, the netplan-directed nameserver 
is set to 127.0.0.1, and stubby is configured to provide encrypted DNS over TLS. 


Of course, Stubby is configured to use appropriate DNS servers that will support 
DNS over TLS service. 


Again, thanks for the article! 
—Jon 
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Shawn Powers replies: |’m glad the DNS bit helped! | recently found a glitch 
(feature? LOL) with my Ubuntu MATE desktop install. | followed my own article, 
because it’s not the sort of thing you do often enough to remember all the details, 
and | discovered that while | had to use “DNS” in the NetworkManager.conf file to 
make it work when | wrote the article, | actually had to use “dns” in my current setup. 
Whoever invented capital letters should be slugged in the arm. 


ll have to check out Stubby. The only other DNS tool I’ve used on the desktop is 
DNSMasq, which I’ve also written about extensively. Stubby is new to me, and one 
more thing I’ll have to investigate! Thanks! 


Re: Dave Taylor and Roman Numerals 

| always enjoy Dave Taylor’s immensely useful Bash column, but | think his latest 
series over-complicates the problem (see Part |, Part Il and Part Ill of Dave’s Roman 
Numerals and Bash series). 


Converting to Roman numerals can be done by taking each of the one- and two-letter 
values you list and repeatedly subtracting them in one go—no need to repeat the 
whole table. This is effectively the “greedy algorithm” approach to change-making. 


| also dislike having to use lots of similar if-blocks, so I'd rather jump through some 
regex hoops to avoid it. See my suggestion below for a more succinct solution: 


#!/bin/bash 
# 
# toroman 
# 
DECVALUE="$1" 
ROMANVALUE="" 
TABLE="M=1000 CM=900 D=500 CD=400 C=100 XC=90" 
TABLE+="_ L=5@ XL=4@ X=10@ IX=9 V=5 IV=4 I=1" 
for T in $TABLE ; do 
R=${T/=*/} ; D=${T/x=/} 
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while [[ $DECVALUE -ge $D ]]; do 
ROMANVALUE+=$R 
let DECVALUE-=$D 
done 
done 
echo $ROMANVALUE 


—Mike 


Dave Taylor replies: Thanks for your note, Mike. | have to admit that there’s 
something about the world of regular expressions that is still daunting. And trying 

to explain a complicated regex to someone who isn’t familiar with the concept (my 
primary reader) is more than a bit overwhelming. Still, it’s a graceful solution. Where 
were you when | was a compsci undergrad at UCSD? 


Possible Article Suggestion 

Regarding Kyle Rankin’s “Back to Basics: Sort and Uniq” from the January 2019 issue: 
lama relative newbie to Linux, about five years or so, on my personal computer at 
home, so | enjoyed this article on going back to basics. Understanding commands and 
how to use them well is how one gets more comfortable with the system. 


| was thinking an article on how to read man pages might be a good idea. | think | 
may have read something on it in the past, but understanding the basics and how to 
understand fully how to run commands might help others too. 


—Patrick Donahue 

Bad Language in the Kernel Code 

Ricky Gervais put it succinctly: “Nobody has the RIGHT to NOT be offended” 
(my emphasis). 


Yes, if you write a comment that says “xyz is a fucking idiot”—it’s probably 
inappropriate to attack someone explicitly; that’s basically just rude. On the other 
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hand, if you write a comment that says, “This chunk of code is fucking awful” Cand you 
can say why), then why not? They’re just words to emphasize your opinion of the code. 


Complaints about bad language are a red herring. The real issue is confusing who 
people are (male, female, black, white, gay, other, etc.) with what they believe 
(homophobic, or not, religious, atheist, pro-life, pro-choice, Republican, Democrat, 


perl is a good thing, etc.). Now everyone, please, get a fucking life! 


—David Jameson 


The Security Issue 

There was a great deal of good information in the February 2019 issue. As a retired 
individual having a bit more time than | used to have, | have taken on the task of 
removing everything | can from “the cloud”. My reasons are not that important, but 
suffice it to say that | dislike any of my data being anywhere but on my hardware 
unless | specifically send it to someone/some-group. At the same time, | want access 
to my data on all my devices regardless of my location. Thus, | established my own 
cloud using Nextcloud. Doing so can be a daunting task once the rather simple initial 


setup is complete. Security is of great importance. 


There are a number of basic tasks in the February 2019 issue that | have not taken on, 
but that | should. So thank you much for an excellent selection of applicable articles! 


As aside note: Passman, which is an add-on to Nextcloud does a pretty darn good job 
of handling passwords. There is a plugin for browsers that works quite well connecting 
to and retrieving passwords from the encrypted database on the server. Also, there is 
an app for mobile devices in beta as well. 


Finally, “Open Science, Open Source and R” by Andy Wills is very pertinent from my 
perspective, as it addresses a rather disturbing problem of the real and/or perceived 
veracity Cor lack thereof) of some research and the broader problem of the recent attacks 
on science. Only the scientific community can address this issue, and the author does a 
good job of showing one way in which that can be done. It is critically important that we 
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re-establish our confidence in science and scientific methodology, and doing so means the 
scientific community is going to have to re-dedicate itself to a higher level of discipline. 


| also want to mention that | agree fully with Glyn Moody (see “If Software Is Funded 
from a Public Source, Its Code Should Be Open Source”). Well said! 


And more generally, as usual, Linux Journal delivers! Well done! 
—Wesley J. Wieland 


From Social Media 
In Response to Receiving a Free Issue of LJ: 


Danny Hernandez @danny13_33: Aww Yeah! Thank you @linuxjournal 
for the subscription to your magazine! Just a quick look and there are tons of 


useful articles that | can apply to my everyday SysAdmining. #Linux #DevOps 
#LinuxJournal #SysAdmin 


In Response to Bryan Lunduke’s “Why Is Linux Spelled Incorrectly”’: 


Steeve McCauley: oh god no, that’s worse than linux. | used to think it 
should be linix, but | gave that up 20+ years ago. 


Jason Lisonbee: |t’s possible the person who first called it Linux thought of your 
spelling but the “u” made it more clear that the commands and behavior were 
intended to be naturally familiar to Unix users. ‘Ux can be short hand for ‘unix’. 
Rob Kingsboro: GNU+LINICS 


Chris Franz: “Linux” would be too close to Minix for Torvalds’ taste, | suspect. 


Still surprises me that there is actual audio of how the man himself pronounces 
it—he’s the foremost authority given that it’s based on his own damn name— 
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and people still intentionally pronounce it LIE-nux. 

Jason Lisonbee: ||RC someone else mashed the name Linus with Unix to call 
it Linux. Just like someone other than Linus witnessed or learned that he had 
some incident with a penguin and made that the mascot; then or later calling it 
Tux, which is now the name of some derivatives, including probably a distro and 
a filesystem, which AFAIK are distinct and separate products/projects. 

Mihalej Archnalej Korcak: it’s LINUKS! 


Abdullah Leghari: |n Asia it’s pronounced like Lienuks or Lyenuks. 


Rob Kingsboro: Well, that was interesting. Arbitrarily putting Xs in things is corny. 


SEND LJ A LETTER We'd love to hear your feedback on the magazine and specific 
articles. Please write us here or send email to Ijeditor@linuxjournal.com. 


PHOTOS Send your Linux-related photos to Ijeditor@linuxjournal.com, and we'll 
publish the best ones here. 


22 | March 2019 | https://www.linuxjournal.com 


~ 99 Gk 


~ XN 


1800+ attendees * 100+ sessions * 40+ vendors 
Open hardware * Social events « Job fair * World famous raffle 
All free to attend 


itcps:/7 


a 


LinuxFest Northwest (est. 2000), an annual Open Source event 
co-produced by Bellingham Linux Users Group and the 
; Information Technology department at BTC. LFNW features 
TECHNICAL |g aia presentations and exhibits on free and open source topics, as 


Bellingham 


COLLEGE aeaennne as ; 
well as Linux distributions & applications, InfoSec, and privacy; 


something for everyone from the novice to the professional! 
Linux User Group 


) https://blog.lfnw.org linuxfestnorthwest YW @lfnw GE linuxfestnorthwest SM info@linuxfestnorthwest.org 


FOSS Project 
Spotlight: Daylight 
Linux Version 3 


Daylight Linux is the only official distribution for the Raspberry Pi to work with the 


Fluxbox interface. With Fluxbox, Daylight Linux is one of the lightest and fastest 
distributions for all Raspberry Pi models. 


Many programs, games and system tools were developed during a long year of work 
in Python 3 to create version 3. 
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Figure 1. 
The System 
at Boot 
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OB /4.00GiB - 0% 


Networking 
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Figure 2. 
The Daylight 
Linux Menu 
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File systems: 
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Temperatures: 

CPU: C - MB: C 


Name PID CPU% MEM% 
. 630 0.50 0.22 
gd 667 0.00 0.08 
Password Generator 601 0.00 0.29 
PyConnect sage 
Package manager 443 0.00 1.26 
System load 361 0.00 0.50 
volume nonitor 462 0.00 0.45 
Resolution 
Virtual Keyboard . 443 0.00 1.26 
wicd 361 0.00 0.50 
wicd-monitor 462 0.00 0.45 
fluxbox 601 0.00 0.29 
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Figure 3. 
The 
Daylight 
Linux 
Desktop 
with System 
Information 


Name PID CPU% MEM% 
Xorg 443 0.50 1.45 
conky 630 0.50 0.22 
dconf-service 712 0.00 0.13 
Mem usage 

Xorg 443 0.50 1.45 
wicd 361 0.00 0.50 
wicd-monitor 462 0.00 0.45 


Xorg 443 0.50 1.45 
wicd 361 0.00 0.50, 
wicd-monitor 462 0.00 0.45 
fluxbox 601 0.00 0.29 


Figure 4. 
The 
Daylight 
Linux File 
Manager 


Desktop Documents 


e PID CPU% MEM% 
Pictures 443 0.50 1.51 
y 630 0.50 0.22 
iemd-hostnam 897 0.00 0.05 
usage “] 
443 0.50 1.51 
847 0.00 1.39 
857 0.00 0.51 


443 0.50 1.51 
nautilus 847 0.00 1.39 
tracker-store 857 0.00 0.51 
wicd 361 0. 0.50 
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The system works with autologin, but you also can use these login/passwords: 
“root’/toor” and “Daylight”/’toor”. 


A live version also is available for computers. This version aims to provide Debian- 
based Linux with the lightness of Daylight Linux. 


Daylight Linux version 3 runs on all Raspberry Pi models, and it’s based on Debian 
Buster. Visit the official website for more information and to download. 


—Hamdy Abou El Anein 
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Antennas In Linux 


For this article, | want to introduce a piece of software I’ve actually used recently in 
my own work. My new day job involves studying the ionosphere using an instrument 
called an ionosonde. This device is basically a giant radio transmitter that bounces 
radio waves off the ionosphere to see its structure and composition. Obviously, an 
important part of this is knowing the radiation pattern of the various transmitters and 
receivers. 


Several methods exist for modeling the electromagnetic fields around conductors, but 
here I’m covering one called NEC2 (Numerical Electromagnetics Code). It originally 
was developed in FORTRAN at the Lawrence Livermore National Laboratory in the 
1970s. Since then, it’s been re-implemented several times in various languages. 


Specifically, let’s look at xnec2c. This package implements NEC2 in C, and it also 
provides a GTK front end for interacting with the core engine. 


X xnec2c 3.6.1-beta ~ x ‘i . 
File Help Figure 1. Launching 
Structure Geometry xnec2c gives you 


a pretty boring 
starting point. 
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xnec2c should be available in most Linux distributions. In Debian-based distributions, 
you can install it with the command: 


sudo apt-get install xnec2c 


Once it’s installed, you can start it with xnec2c. The default display doesn’t show 
anything until you actually start using it. 


X xnec2c 3.6.1-beta — O x 
File View Help Currents Charges Frequency Loop Step -l 
| 0.00 = ----- dB 140.000 — + |MHz |+ Apply 
X Y 7 45 =— | + 45 —- +  Zoom%) 100 —-|+ 


Structure Geometry 


Figure 2. Loading an input file, you begin with a geometric view of the relevant antenna 
wires, other conductors and any ground planes. 
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xnec2c’s history still affects how it behaves to the present day. This is most clear when 
you look at the input file’s format. The basic structure is based on the idea of a punch 
card, where each “command” to xnec2c is given by a command card—a definite 
holdover from its FORTRAN roots. Luckily, the GTK front end to xnec2c provides a 
reasonably functional way of building up these input files. 


C. 


Several example files should be available with your installation of xnec2c. In my Ubuntu 


distribution, they’re located in /usr/share/doc/xnec2c/examples. These input files have 
a filename ending of “.nec”. Select one as a starting off point to play with xnec2c, and 
then go ahead and make the required alterations necessary for your own project. 


x Xnec2c: Radiation Patterns 


File View Gain Pattern E/H Fields Frequency Loop 


-.coo . 3.28 dB 140.000 


x Y Z 17 - + i 17 


Radiation Pattern: - Total Gain - Linear Power 


Si 
SS lied 
SS 
am | | || 


Figure 3. You easily can model the radiation pattern from an antenna system. 
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x Xnec2c: Radiation Patterns 
File View Gain Pattern 


0 


Xx hd Z 


» 45 


— Oo x 
E/H Fields Frequency Loop Step 21 
-8.12 dB 140.000 §) |MHz [> Apply i 
» 45 Zoom % 100 


Near Fields: - E Field - H Field - Poynting Vector 


Figure 4. If your input file has the appropriate entries, you can see a display of the electric 


and magnetic fields. 


The central window pane 
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provides a geometric view of the actual antenna structure 


three dimensions. You can click and drag the diagram to rotate the view and see 
rom all angles. There are two larger buttons at the top of the window, named 


ecting them alternately will show either the distribution of 
n of charges caused by the driving current. 


ical layout is correct, you can start to see what the 
of the system looks like. On the menu bar at the top, there 
icking “Radiation Pattern” pops up a new window where 
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you can view what the electromagnetic radiation pattern looks like. A few different 
options are available. The first, “Gain Pattern”, provides what most people probably 
think of as the radiation pattern. 


Again, you can click and drag the image to get a better view of the pattern from 
various angles. 


In the same window, there is a second display option. Assuming your input file has the 
appropriate entries, click the “E/H Fields” button to see a view of the electric and/or 
magnetic fields for your configuration. 


X Xnec2c: Frequency Data Plots = oO x 
File View Max Gain Direct. Viewer VSWR ZrfZi Zm/Zph Freq Loop 
FMHz 150.000 Max Gain 10.6 VSWR, 3.6 Zreal 16.9 Zim, 21.5 Zo 50.0 


Frequency Data Plots - Total Gain 


Max Gain & F/B Ratio vs Frequenc F/B Ratio db 9g 99 


Ee eae el a ee ee 
ae aa | | a ee 
ee ee ee ee ee eee LE | 00 

re ee ee ee ee ee 


21.00 
20.00 
19.00 
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17.00 
16.00 
15.00 


14.00 


13.00 
140.0 141.0 142.0 143.0 144.0 145.0 146.0 147.0 148.0 149.0 150.0 


Figure 5. You can get several detailed plots of the electrical behavior of your antenna design. 
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X Xnec2c: Frequency Data Plots = O x 
File View Max Gain Direct. Viewer VSWR Zr/Zi Zm/Zph Freq Loop 
FMHz 150.000 Max Gain 10.6 VSWR 3.6 Zreal 16.9 Zim, 21.5 Zo, 50.0 


Frequency Data Plots - Total Gain 


Max Gain & F/B Ratio vs Frequenc F/B Ratio db 97.50 


140.0 141.0 142.0 143.0 144.0 145.0 146.0 147.0 148.0 149.0 150.0 
VSWR vs Frequenc 


140.0 141.0 142.0 143.0 144.0 145.0 146.0 147.0 148.0 149.0 150.0 


Figure 6. You can see multiple plots of the electrical behavior stacked on top of each other 
to get a better overall view of your antenna configuration. 


Several options are available at the top of the window. For example, you can 
change the frequency of the driving current to see how your system behaves at 
different frequencies. 


The second major display that’s available shows how your antenna design behaves over 
a series of driving frequencies. Clicking View—Frequency Plots pops up a new window. 


The first button shows you how the gain changes over a given frequency sweep. 
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Clicking on each subsequent button adds a new graph to the same window. For 
example, click the VSWR button to see how the Voltage Standing Wave Ratio (VSWR) 
changes over frequency. 


You probably will need to resize the window if you want to view more than a couple 
graphs simultaneously. As with the E/H Fields plot from earlier, whether these plots 
behave as you expect depends on the input file you’re using. 


The example files provide a great place to start, but they’re not likely to match your 
situation exactly. In those cases, you'll want to edit the input files to configure the 


Manage Cards File Save 
Add Remove Clear Save As Save {¥ Apply Revert 
Comments 
Card Comments 
cM --- NEC2 Input File created or edited by xnec2c 3.5 --- 
cM Yagi for 145 MHz 
CE --- End Comments --- 
Edit Geometry Data 
Wire Arc Helix Patch Cylinder Move Reflect Scale 
Geometry 
Card 11 12 Fl F2 F3 F4 F5 F6 F7 
GW 1 25 0.00000E+00 5.09000E-01 0.00000E+00 0.00000E+00 -5.09000E-01 0.00000E+00 5.00000E-03 
GW 2 25 4.00000E-01 4.84000E-01 0.00000E+00 4.00000E-01 -4.84000E-01 0.00000E+00 5.00000E-03 
GW 3 22 7.00000E-01 4.59000E-01 0.00000E+00 7.00000E-01 -4.59000E-01 0.00000E+00 5.00000E-03 
GW 4 22 1.10000E+00 4.50000E-01 0.00000E+00 1.10000E+00 -4.50000E-01 0.00000E+00 5.00000E-03 
Gw 522.1. 5annne+nn 4 annanr-n1_9 nnAANE+NNn 1 SNNANE+ON. -4 annANe-n1 9. NANANF-+NN_ 5 _NANANF-n3 
Edit Control Commands 
Excitation Frequency Ground (GN) Loading Network Trans. Line 
Radiation Near Fields Ground (GD) Kernel Interaction Execute 
Commands 
Card 11 12 13 14 Fl F2 F3 F F5 F6 
FR Oo 21 0 0 1.40000E+02 5.00000E-01 1.50000E+02 0.00000E+00 0.00000E+00 0.00000E+00 
EX Oo 2 13 0 1.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 
LD 5 0 0 3.70000E+07 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 
NH Oo 0 0 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 0.00000E+00 
NE 0 — Of. - 15 1 -1.40000F+00 -1.40000F+00 _ 5.00000F-0?  ?.00000F-01 _?.00000F-01 _0.00000F+00 


Figure 7. You can use the built-in editor to customize the input file to your personal specifications. 
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system exactly as you need it to be. 


Although you simply can open the input file in any text editor, xnec2c includes an edit 


function built in. Clicking the File—Edit menu item pops open a new window where 
you can see all of the input cards. 


You can change the values for already existent cards, but the best feature of this 
editor is that you can add new cards with the click of a button. If you are new to NEC2 
and its input format, you may not necessarily remember the codes and formats for 
the various types of input cards. The editor window helps with this. There even are 
buttons, like mirror and scale, that aid in the design phase by taking advantage of 
symmetries you may have in your antenna. 


In each of these displays, you have a couple different options when it comes to 


saving your results. The simplest is clicking File—Save As. This lets you save the 
resulting image as a PNG file. If you select File—Save As gnuplot, the actual data 
is exported into a file, and the format is appropriate for the type of data that you 
might want to feed into gnuplot. This way, you can tailor the resulting graphical 
display for your particular situation. 


Results that xnec2c generates are not directly comparable with those from other 
implementations of NEC2. Its focus is more on the functionality of being interactive. 
If you want more traditional output, you can use xnec2c to view your antenna 


configuration, and then use nec2c to generate your final results. 


—Joey Bernard 
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Journal Podcast 
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new digital world: https://www.linuxjournal.com/podcast. 
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Lessons in Vendor 
Lock-in: 3D Printers 


The open nature of the consumer 3D printing industry has made for 
a much more consumer-friendly world. 


This article continues a series that aims to illustrate some of the various problems 
associated with vendor lock-in. In past articles, I’ve given examples showing how 
proprietary systems from disposable razors to messaging apps have replaced more 
open systems leading to vendor lock-in. This article highlights an ecosystem that, so 
far, has largely avoided vendor lock-in and describes the benefits that openness has 
provided members of the community, myself included: 3D printing. 


’ve been involved in 3D printing for several years. ’'ve owned a number of printers, 
and I’ve seen incredible growth in the area from an incredibly geeky fringe to the much 
more accessible hobby that it is today. I’ve also written quite a few articles in Linux 
Journal about 3D printing, including a multi-part series on the current state of 3D 
printing hardware and software (see the Resources section for links to Kyle’s previous 
Linux Journal articles on 3D printing). | even gave a keynote at SCALE 11x on the free 
software and open hardware history of 3D printing and how it mirrors the history of 
the growth of Linux distributions. 


The Birth of 3D Printing in the Hobbyist Market 

One interesting thing about the hobbyist 3D printing market is that it was founded on 
free software and open hardware ideals starting with the RepRap project. The idea 
behind that project was to design a 3D printer from off-the-shelf parts that could 
print as many of its own parts as possible (especially more complex, custom parts like 
gears). Because of this, the first generation of 3D printers were all homemade using 
Arduinos, stepper motors, 3D-printed gears and hardware you could find in the local 
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hardware store. 


As the movement grew, a few individuals started small businesses selling 3D printer 


kits that collected all the hardware plus the 3D printed parts and electronics for you 
to assemble at home. Later, these kits turned into fully assembled and supported 
printers, and after the successful Printrbot kickstarter campaign, the race was on 

to create cheaper and more user-friendly printers with each iteration. Sites like 
Thingiverse and YouMagine allowed people to create and share their designs, so even 
if you didn’t have any design skills yourself, you could download and print everyone 
else’s. These sites even provided the hardware diagrams for some of the more popular 
3D printers. The Free Software ethos was everywhere you looked. 


3D Printing Enters the Consumer Market 
As 3D printers entered the mainstream, and consumer brands started to enter the 


market, | figured the honeymoon was over. Under the guise of ease of use, many 
of these first mainstream efforts bundled their printers with proprietary, branded 
software. Some even went as far as copying the old inkjet printer lock-in models 


and sold their 3D printer hardware at a discount or a loss and made up the cost 


with proprietary filament. Even MakerBot—one of the original 3D printer brands— 
abandoned releasing its designs to the public. 


Of course, not all brands went the proprietary route. Some of the original brands— 
like Ultimaker, Printrbot and Lulzbot—all maintained their commitment to free 
software and open hardware. The pure RepRap hobbyist community also continued 
to improve and publish new designs. | figured if everything else went the proprietary 
route, | would just stick with my Printrbot. 


The fact that most hobbyists stuck with open designs and free software and firmware 


led to some amazing innovations in 3D printing, such as the OctoPrint front end. 
Instead of always connecting your computer to your printer to start a job, OctoPrint 
allowed you to monitor your printer remotely and set up new print jobs all over the 
network via a web interface. OctoPrint even offered an image for Raspberry Pis, 

and that combination became one of the most popular ways to talk to your printer. 
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Thingiverse and YouMagine were full of case designs to let you mount a Raspberry Pi 
and camera to your printer. 


OctoPrint flourishes because most of the printers were based off a standard and 
open hardware and firmware platform. They all spoke G-code—a series of common 
commands that allowed you to move stepper motors, set temperatures and otherwise 
monitor and control your printer. Using slicing software, like Slic3r or Cura, you 
would load an STL file containing your 3D design into the program, decide which 
printer quality settings you wanted to use, and then your slicer would convert the STL 
file into G-code. Normally, you then would take that G-code file and either copy it to 
an SD card in your printer or use a different program to load the G-code into your 
printer and monitor the print (Cura was able to act both as a slicer and as a front 
end). Because Cura was free software though, OctoPrint was able to incorporate 

its slicing engine into its own software, so you could upload STL files directly to the 
OctoPrint web interface, and it would slice them for you. 


Bring in the Clones 
One of the side effects of high-quality free hardware, firmware and software for 3D 


printers being readily available is that it enabled a large number of low-cost clones of 


popular 3D printer designs to flood the market. Where before a lower-cost printer 
was between $600-$1000, and higher-end printers were closer to $2000-$3000, now 
you could find clones of those models for half the price. What’s more, although some 
of those $200 and $300 printers cut corners (and some caught fire!), for the most 
part, they actually produced good quality prints. 


An interesting result of those clones was that since they used free software and 
firmware, they all worked with the same free software tools everyone already was 
using. In fact, some of them even released their own Cura profiles or complete forks 
of Cura with their own printer’s settings enabled by default. The quality of the free 
software tools made it so that supporting them became critical in many consumers’ 
decision-making process. Many of the proprietary consumer models couldn’t 
compete, especially those that were incompatible with free software tools. 
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Oh Snap 


Of course, those clones had another side effect. They made it much more difficult for 
some of the established brands to compete. My personal favorite brand, Printrbot, 
announced it was filing for bankruptcy this past fall. The next day, while working on 
my printer, | managed to snap the micro-USB port off the motherboard. Normally, I’d 
just order a replacement board, but of course, now that wasn’t an option. Yet, thanks 
to the open standards in 3D printing, there were a number of generic third-party 
RepRap boards | could replace it with. Sure, it would mean spending quite a bit of time 
with the board to connect electronics, calibrate things and flash firmware, but the 


point is that it was an option at all (and in general, it’s an option for any average 3D 
printer on the market). 


Unfortunately, | didn’t really have the free time to invest in such a project, so instead, 
my 3D printer sat in its broken state. Fortunately, | was surprised on Christmas with a 
new 3D printer—one of those low-cost clones. Because of all of the open standards 
in this industry, | was able to connect it directly to my existing OctoPrint server, tell 
the server about my printer’s dimensions and settings, and get right back to printing. 
So far, ’ve been pleased and surprised with the quality of the prints, and | understand 
that without the open hardware and free software firmware and tools, it would have 


been a much different story. 

The moral here is that although there sometimes are risks in a world without 
vendor lock-in, that world is much better for the end user. Open standards and 
interoperability provide so many more options than you would have in a proprietary 


world, including giving you options even if your vendor closes shop. 


—Kyle Rankin 
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“Lessons in Vendor Lock-in: Shaving” by Kyle Rankin 


“Lessons in Vendor Lock-in: Messaging” by Kyle Rankin 


“Getting Started with 3D Printing, the Software” by Kyle Rankin 


“Getting Started with 3D Printing, the Hardware” by Kyle Rankin 


“What’s New in 3D Printing, Part I: Introduction” by Kyle Rankin 


“What’s New in 3D Printing, Part Il: the Hardware” by Kyle Rankin 


“What’s New in 3D Printing, Part Ill: the Software” by Kyle Rankin 


“What’s New in 3D Printing, Part IV: OctoPrint” by Kyle Rankin 


RepRap.org 
Thingiverse 
YouMagine 

MakerBot 

Ultimaker 

Lulzbot 

Printrbot (Wikipedia) 
OctoPrint 

G-code 

Slic3r 


Cura 
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* Groboards has launched a new “tiny, Adafruit Feather form-factor ‘Giant Board’ 


SBC that runs Linux on Microchip’s SiP implementation of its Cortex-A5-based 
SAMASD SoC and offers 128MB RAM, micro-USB, microSD and |/O including ADC 
and PWM”, Linux Gizmos reports. There’s no pricing or availability information 
at the time of this writing, but see the OSH Park blog and the Groboards site for 


specs and more info. 


- Anew open-source hardware project called Alias will keep Amazon and Google 
smart assistants from spying on you. According to the project’s GitHub page, 
“Alias is a teachable ‘parasite’ that is designed to give users more control 
over their smart assistants, both when it comes to customisation and privacy. 
Through a simple app the user can train Alias to react on a custom wake-word/ 
sound, and once trained, Alias can take control over your home assistant by 
activating it for you.” 


- Anew major release of the open-source Metasploit Framework is now available. 
According to the Rapid7 blog post, version 5.0 of the penetration-testing tool 
is the first milestone update since version 4.0 came out in 2011. Along with a 
new release cadence, “Metasploit’s new database and automation APIs, evasion 
modules and libraries, expanded language support, improved performance, and 
ease-of-use lay the groundwork for better teamwork capabilities, tool integration, 
and exploitation at scale.” 


* Orange Pi 3 SBC is now available. Linux Gizmos reports that the open-source 
hardware platform, Allwinner H6-based Orange Pi 3 SBC is now available for 
$30, or for $40 with 2GB of RAM and 8GB eMMC. Also, other highlights include 
“GbE, HDMI 2.0, 4x USB 3.0, WiFi-ac, and mini-PCle.” For more info, visit the 
Orange Pi 3 AliExpress page. 
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Inkscape is finally reaching the 1.0 milestone after 15 years of development. 
Softpedia News reports that Inkscape 1.0 will feature “an updated user interface 
that offers better support for 4K/HIDPI screens and theming support, the ability to 
rotate and mirror canvases, new options for exporting to the PNG image format, 
variable fonts (requires pango 1.41.1 or higher), as well as much faster path 
operations and deselection of a large amounts of paths.” You can download the 
pre-release alpha as an Appimage from here and see the release notes here. 


Creative Commons recently announced that 30,000 high-quality digital images 
from the Cleveland Museum of Art are now available. The free and open digital 
images are now under the CCO and available via their API. The “CCO allows anyone 
to use, re-use, and remix a work without restriction.” Museum Director William M. 
Griswold said “Open Access with Creative Commons will provide countless new 
opportunities to engage with works of art in our collection. With this move, we 
have transformed not only access to the CMA’s collection, but also its usability— 
inside as well as outside the walls of our museum.” 


Dell launched its new XPS 13 9380 Developer Edition laptop, which runs Ubuntu 
out of the box. According to Forbes, highlights include Intel 8th generation i3, 
iS and i7 processors; Ubuntu 18.04 LTS preloaded; InfinityEdge display with top 
camera placement; and much more. See Dell.com for more information. 


Raspberry Pi announces its Computer Module 3+ (CM3+) is now available 

for $25. The CM3+ is the “newest version of our flexible board for industrial 
applications and offers over ten times the ARM performance, twice the RAM 
capacity, and up to eight times the Flash capacity of the original Compute Module.” 
The company also has released a refreshed Compute Module Development kit. 
The CM3+ will be available until at least January 2026. 


Firefox 65.0 was released to Channel users. New features include enhanced 
tracking protection, better experience for multilingual users, support for 
HandOff on macOS, better video streaming for Windows users, and improved 
performance and web compatibility, with support for the WebP image format. 
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Go here to download Firefox. 


Lucern Custom Instruments from the UK teamed up with Tracktion Corporation 

of Seattle to create Spirit Animal, an electric guitar with a Raspberry Pi synthesizer 
built in. According to the Raspberry Pi Blog, the guitar “boasts an onboard Li-ion 
battery granting about 8 hours of play time, and a standard 1/4” audio jack for 
connecting to an amp. To permit screen-sharing, updates, and control via SSH, the 
guitar allows access to the Pi’s Ethernet port and wireless functionality.” See also 
the Gear News website and the Lucern Instruments Facebook page for more 
information. 


Kodi 18.0 “Leia” is now available for all supported platforms. This is a major 
release, reflecting nearly 10,000 commits, 9,000 changed files and half a million 
lines of code added. This new release features support for gaming emulators, 
ROMs and controls; DRM decryption support; significant improvements to the 
music library; live TV improvements; and much more. See the changelog for more 
details, and go here to download. 


ZaReason debuted its new Gamerbox 9400, “the ultimate Linux gaming PC”. 
And, the Gamerbox is just the beginning, ZDNet reports, quoting ZaReason CEO 
Cathy Malmrose: “Our current team is mostly gamers so, not surprisingly, that is 
the direction we are going. We have a full line of gaming machines in R&D.” The 
Gamerbox runs Ubuntu 18.04, with a 64-bit Pentium 3.8Ghz G5500 Coffee Lake 
processor and 8GB of DDR4 memory. 


Google announced two new audio apps for Android to help people who are 
deaf or hard of hearing: Live Transcribe and Sound Amplifier. Live Transcribe 
“takes real-world speech and turns it into real-time captions using just the phone’s 
microphone”. Live Transcribe will roll out gradually as a limited beta via the Play 
Store and pre-installed on Pixel 3 devices. You can sign up here to be notified 
when it’s more widely available. Sound Amplifier makes “audio more clear and 
easier to hear. You can use Sound Amplifier on your Android smartphone with 
wired headphones to filter, augment and amplify the sounds in your environment. 
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It works by increasing quiet sounds, while not over-boosting loud sounds.” Sound 
Amplifier is available now via the Play Store and supports Android 9 Pie or later and 
comes pre-installed on Pixel 3. 


The Document Foundation announced the official release of LibreOffice 

6.2 with NotebookBar. This is a major new release that “features a radical new 
approach to the user interface—based on the MUFFIN concept—and provides 
user experience options capable of satisfying all users’ preferences, while 
leveraging all screen sizes in the best way.” This version has many new features, 
including substantial changes to icon themes, context menus are tidied up and 
interoperability with proprietary file formats has been improved. See this video 
for details on all the new features. Note that LibreOffice 6.1.5 also was released 
recently for enterprise-class deployments. You can download LibreOffice 6.2 or 
LibreOffice 6.1.5 from here. 


Raspberry Pi has opened a store in the Grand Arcade, Cambridge, UK. See this 
video for details and follow #RPiStore for more photos and info. 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email ljeditor@linuxjournal.com. 
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LibrePlanet brings together free software enthusiasts and 
professionals to learn skills, celebrate free software 
accomplishments, and face challenges to software freedom. 


Featuring keynote speeches by: 


* Bdale Garbee - Free software activist, Debian Project contributor 

* Micky Metts - Free software activist, co-owner of Agaric tech cooperative 
* Tarek Loubani - Physician & founder, Glia Project 

* Richard Stallman - President & founder, Free Software Foundation 


Downsides to 
Raspberry P! 
Alternatives 


Learn about some of the risks when choosing an 
alternative to a Raspberry Pi for your project. 


By Kyle Rankin 


| have a lot of low-cost single-board computers (SBCs) 

at my house. And, I’ve written a number of articles for 
Linux Journal that discuss how | put those computers to 
use—whether it’s controlling my beer fridge, replacing a 
rackmount file server, acting as a media PC connected to 
my TV or as an off-site backup server in my RV (plus many 
more). Even more recently, | wrote a “Pi-ventory” article 
where | tried to count up just how many of these machines 
| had in my home. 


Although the majority of the SBCs | use are some form of 
Raspberry Pi, | also sometimes use Pi alternatives—SBCs 
that mimic the Raspberry Pi while also offering expanded 


features—whether that’s gigabit Ethernet, faster CPUs, SATA 


ports, USB3 support or any number of other improvements. 
These boards often even mimic the Raspberry Pi by having 
“Pi” in their names, so you have Orange Pi and Banana Pi 
among others. Although Pi alternatives allow you to solve 


some problems better than a Raspberry Pi, and in many cases 
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they provide hardware with better specifications for the same price, they aren’t 
without their drawbacks. So in this article, | take a look at the downsides of going 
with a Pi alternative based on my personal experience. 


Third-Party Support 


The initial Raspberry Pi was a runaway success, and all of the subsequent models 
have sold incredibly well. There are only a few variants on the Raspberry Pi 
platform, and later hardware upgrades have done a good job at maintaining 
backward-compatibility where possible (in particular with overall board dimensions 
and placement of ports). There also have been only a few “official” Raspberry Pi 
peripherals through the years (the camera being the best example). When you 
have this many of a particular hardware device out in the world, and the primary 
vendor is mostly focused on the hardware itself, you have a strong market for add- 
ons and peripherals from third parties. 


The secondary Raspberry Pi market is full of cases, peripherals and add-on hardware 
like USB WiFi dongles that promise to be compatible out of the box with earlier 
models that didn’t include WiFi. Adafruit is a good example of an electronics vendor 
who has jumped into the Raspberry Pi secondary market with a lot of different 
hobbyist kits that feature the Raspberry Pi as the core computing and electronics 
platform. That company and others also have created custom add-on shields intended 
to stack on top of the Raspberry Pi and add additional features including a number of 
different screen options, sensors and even cellular support. There’s even a company 
that offers a case to turn a Raspberry Pi into a small laptop. 


By contrast, the secondary market for Pi alternatives is pretty small. You might find 
cases or other basic peripherals, but because there just aren’t as many of those 
devices out in the world, it’s much riskier for a company to go to the trouble of 
making specific add-ons just for those devices. Fortunately, you often can re-use 
peripherals intended for a Raspberry Pi in these Pi alternatives. Many competitors 
make a point to ensure they are pin-for-pin compatible with the Raspberry Pi’s GPIO 
pins, which are used for many add-ons. Even with this though, compatibility is not a 
sure thing—you have to do your research in many cases or risk wasting money on an 
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add-on that might fit but might not have software support once you boot. 


The Community 

One of the major things Raspberry Pi has going for it is its community. Because 

of its popularity and how many of these devices were sold compared to those of 
competitors, it has an enormous community of users that spread across all kinds of 
disciplines—from electrical engineers to gamers to sysadmins to artists. 


There are a lot of advantages to having a large and diverse community behind a 
project. It means that when you want to do something with your SBC, you are less 
likely to be the first. There’s a good chance you will find more complete (and more 
up-to-date) documentation and HOWTOs for whatever you want to do. It also means 
if you run into problems following one of these guides, there is a community on 
forums and chat rooms who can help walk you through it. 


Many of the Pi alternatives do have communities in their own right with forums and 
chat rooms that you can turn to for support. There are also guides online for some 
of the more common use cases for a particular board (for instance, boards that 
feature SATA and gigabit Ethernet being used as a file server). Some of the common 
software projects (like media PC projects) often also include Pi alternatives in their 
documentation alongside Raspberry Pi if a particular board becomes popular for 
that use. So in many cases, the community is there, but if you start to stray from the 
common uses for your particular board, you are more likely to be on your own, or at 
least you'll have to adapt a Raspberry Pi guide to your SBC. 


OS Support 


There are a couple official operating systems for the Raspberry Pi direct from the 


vendor. Due to the popularity of the Raspberry Pi and the number of people using 
them, these operating systems support the hardware well and receive regular 
updates. Raspbian, one of the most popular OS options for the Raspberry Pi, 
essentially behaves like a standard Debian distribution with Raspberry Pi-specific 
scripts and kernel patches so you can get the most out of the hardware (and perform 
overclocking and other tweaks). 
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The community also has created a wide range of special-purpose OSes for the 
Raspberry Pi based on Raspbian that add specific features. Some examples include 
OSMC, which adds the Kodi media PC software, and OctoPrint, which adds a 
network-based 3D printer front end. There are also projects that build a custom 
Raspberry Pi OS from the ground up, such as OpenELEC and LibreELEC, which 
include Kodi just like OSMC does but with a stripped-down, special-purpose OS that 
boots much faster. 


OS support, in particular kernel support, is probably the biggest downside to 
choosing a Pi alternative. Each vendor tends to provide a few different OS options 
with their boards, some that mimic Raspbian and others that are based on Android. 
These distributions also provide custom kernels with patches to support the 
hardware, but unfortunately compared to the Raspberry Pi world, it’s a toss-up 

as to how well these kernels support the hardware. I’ve had some Pi alternatives 
work perfectly well and stably with the vendor-provided Linux OS, while others 
seemed incredibly unstable. These custom kernels also are problematic because in 
some cases, it can be challenging to track down the source code for their custom 
patches. Even if you do find the patches, you might be on your own if you want to 
apply them to a newer kernel. 


The other problem with Pi alternatives is fragmentation and long-term support for 
the provided OS. Vendors tend to create their own custom distributions for their 
boards instead of using Raspbian or other shared distributions. These vendors also 
tend to create updated models every few years, but the OS for a particular revision 
tends to freeze in time. This means outdated Linux distributions and Android 
versions as time goes on. If you are someone who wants to use these SBCs as a 
server platform in particular, having regular security updates and overall updates 
for your services is very important. In these cases, the kernel packages also tend to 
freeze in place. 


Fortunately, the Armbian project has stepped up to provide a general-purpose 
Raspbian-like OS for Pi alternatives. Depending on your board, Armbian will have 


Debian- and Ubuntu-based images you can use, each with kernels patched to support 
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your hardware. Unfortunately, for some boards, the kernel is a bit less stable, and 
Armbian tries to gauge stability for all of the hardware it supports. Armbian also gets 
regular updates, so a board that was unstable one day might improve over time. 
More important, you will get more regular updates to the rest of the software on 
your system with less worry that the device will be abandoned. As we all accumulate 
more and more of these computers around our homes, regular security updates are 
incredibly important. 


Conclusion 

If you are debating between a Raspberry Pi and another board for your project, | hope 
this article has been helpful. In general, my advice is just to do your research before 
you buy any particular board—even a Raspberry Pi. Without proper research, you 
may end up with a board that won’t work well for your project or that works only with 
an ancient OS. Defaulting to a Raspberry Pi is also not a silver bullet. ’ve chosen Pi 


alternatives for many of my projects specifically because a Raspberry Pi would have 
been under-powered (or didn’t have proper hardware support) for my needs. 


Send comments or feedback 
via http://www.linuxjournal.com/contact 
or email ljeditor@linuxjournal.com. 
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Resources 


- “Temper Temper” by Kyle Rankin 


“Temper Pi” by Kyle Rankin 

“Working on My Temper” by Kyle Rankin 

“N900 with a Slice of Raspberry Pi” by Kyle Rankin 

“Raspberry Strudel: My Raspberry Pi in Austria” by Kyle Rankin 
“Flash ROMs with a Raspberry Pi” by Kyle Rankin 


“LJ Tech Editor’s Personal Stash of Raspberry Pis and Other Single Board 
Computers” by Kyle Rankin 


“Papa’s Got a Brand New NAS” by Kyle Rankin 

“Banana Backups” by Kyle Rankin 

“Super Pi Brothers” by Kyle Rankin 

“DIY RV Offsite Backup and Media Server” by Kyle Rankin 
“Two Portable DIY Retro Gaming Consoles” by Kyle Rankin 
“Raspberry Pi Alternatives” by Kyle Rankin 

Raspbian 

OSMC 

Kodi 

OctoPrint 

OpenELEC 

LibreEEEC 

Armbian 


Adafruit 
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By Reuven M. Lerner 


Like many other nerds, | love word puzzles. I’m not always great 
at them, and | don’t always have time to do them, but when | 
do, | really enjoy them. 


| recently discovered a new daily puzzle, known as “spelling 
bee”, that the New York Times offers online. The idea is simple. 
There are seven different letters, one in the center of a circle 
and six around it. Your job is to make as many different words 
as you can from those seven letters. Each word must be at least 
four letters long, and each word also must contain the center 
letter. You can use each letter as many times as you want. 

So if the letters are “eoncylt”, with a center letter of “y”, 
some of the words you could create might be “cyclone”, 
“eyelet” and “nylon”, 


The online game gives you a score based on how many words 
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you’ve made from the potential pool. If you get them all, you’re awarded “queen 
bee” status. 


| do pretty well at this puzzle, but I’ve never managed to find all of the hidden words. 
Nevertheless, | have become queen bee on a few occasions. How? The answer is 
simple. | cheated. How? Using Python, of course. 


Now, cheating at games isn’t necessarily the first order of business when it comes to 
programming. And cheating at word games in which you’re competing against yourself 
is probably a sign of unhealthy competition. But, doing so also provides a great way 
to review some of the ways you can use Python’s built-in data types and the ease with 
which you can process words and text. 


So in this article, | explore a number of ways you can cheat—and yes, become the 
queen bee, if only for a day. 


Trying All Combinations 

To start, you simply might try to form all of the possible combinations you can 

with the letters you’re given. As you might remember from high-school math class, 
there’s a difference between “permutations” and “combinations”. When you generate 
“permutations”, the order is important, but when you generate “combinations”, the 
order is not important. 


You easily can see this using Python’s itertools module, a part of the standard 
library that has functions named permutations and combinations. Each takes 
both an iterable data structure and the number of items you want in each resulting 


list. For example: 
>>> List(itertools.combinations(['a', 'b', ' 
[Ven ‘bh Jy (“a", Fe ie (ta a ee Con" 1 
Uy ae ‘d*)] 


>>> List(itertools.permutations(['a', 'b', 'c', 'd'], 2)) 
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(‘ety ta"), 
Cbs 7A"); 
b> "es; 
bs a?) 
(te*, tad, 
ee. Os 
Ce, a"); 
("dt tat); 
(40° 5. "O° Js 
Gas Lara ou oD | 


As you can see, the output from combinations saw the order as unimportant and 
thus returned just ('a', 'b'). But permutations saw the order as important, and 
thus returned both ('a', 'b') and('b', ‘a'). 


You could use this to generate all of the letter combinations for possible words 

and then sift through that, right? Well, not really, for two reasons. First, the game 
allows you to repeat letters. And second, these functions let you specify a only single 
number of outputs. 


You can solve the first problem by using the combinations_with_replacement 
function, which not only has along name, but (as the name states) also allows letters 
to appear more than once in the output. For example: 


>>> List(itertools.combinations_with_replacement(['a', 'b', 
etic’, or ale 9)) 
[C"a", at); 


Oats." )s 
ag. (OD 
Ca" °C") 
(be 5. “8? Jy 
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However, you want to find words that are at least four letters long. Let’s assume 

that the longest possible word would be 12 letters long. You could use a for loop, 
appending the results from each iteration to a list. But an even more Pythonic 

way to do this would be to use a list comprehension—or even better, a nested list 
comprehension. Comprehensions are, in my experience, one of the hardest concepts 
for new Python developers to use. However, they are perfect for creating and 
transforming sequences, which is precisely what’s happening here: 


>>> one_combination 
for n in range(4, 13) 
for one_combination in 
itertools.combinations_with_replacement('abc', n)] 


This code iterates over the range from 4 to 13, thus producing the integers from 4 
through 12. For each of these values of n, you then produce all of the combinations, 
with replacement, of that length and with the letters “a”, “b” and “c”. The results then 
are output as a list. 


This is great, but it’s missing at least two things. First, you aren’t interested in 
combinations, so much as words. And there aren't just a few letters for which yourre 
searching, but seven letters—one of which must appear in the word. 


So let’s beef up the comprehension a bit in order to get all of the words: 
>>> all_letters = ‘eoncylt' 
>>> center_letter = ‘'y 


>>> ['', join(one_combination) 
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for n in range(4, 13) 

for one_combination in 
itertools.combinations_with_replacement(all_letters, n) 
if center_letter in one_combination] 


The good news is that this now indeed has generated all of the possible combinations 
that might give you the answer. The bad news is that it’s also generated a lot of 
combinations that aren’t really words. Indeed, according to my count, this created 
31,788 “words”, including such greats as “occccccylt” and “eeeyt”. 


So yes, you could become the queen bee by going through and entering all of these 
words, one by one, as input into the game. Somehow though, | think entering 31,788 
words takes the fun out of cheating. 


Enter the Dictionary 

To make cheating more efficient and fun, let’s try a different strategy. Instead of 
generating all of the possible combinations of letters, let’s instead search through 
only those combinations that are correct. How can you know what’s correct? Via 
the dictionary, of course—and the fact that Linux comes with an English-language 
dictionary makes this easier. 


Indeed, although it often can be useful to generate combinations, it’s probably a wiser 
strategy just to start with the dictionary and choose the words that fit your needs. 


The dictionary that I’m using for this example is in /usr/share/dict/american-english, 
and it contains 102,401 different words, each on a line by itself. The fact that each 
word is on a separate line turns out to be a great advantage, because it means that 
you can (once again) create a list comprehension. In this case, the source of the list 
comprehension won't be an iterator from itertools, but rather the dictionary file 
itself. Iterating over a file in Python gives you one line per iteration. Here’s how you 
can get these words: 


[one_word.strip() 
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for one_word in open(words_file)] 


But, wait a moment. This will return all of the words. You're interested only in those 
words that contain the seven letters in the puzzle, as well as the center letter. 


One solution to this would be to write a function that checks whether the word fits 
your needs. For example 


>>> def is_legal(word, all_letters, center_letter): 
word = word.strip() 


if len(word) < 4: 
return False 


if center_letter not in word: 
return False 


for one_letter in word: 
if one_letter not in all_letters: 
return False 
return True 


The function would appear to work too: 


>>> is_legal('cy', all_letters, center_letter) 
False 


>>> is_legal('cycle', all_letters, center_letter) 
True 


>>> is_legal('hairbrush', all_letters, center_letter) 
False 
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Armed with this function, you now can read through the dictionary and get the 
words that fit: 


[one_word.strip() 
for one_word in open(words_file) 


if is_legal(one_word, all_letters, center_letter) ] 


Note the use of the strip method to remove leading and trailing whitespace from 


the word, mostly because of the newline that you'll get from each word in the file. For 
this reason, you'll also use strip in the is_legal function to ensure that you don’t 
have to deal with the newline. 


So, does it work? The answer is yes, mostly. On some days, my Python program finds 
words that weren’t in the game. And on other days, the game is looking for words that 
aren't in the Linux dictionary. But for the most part, everything seems to work well, 
and although | try hard to win the game each day, there are definitely some days when 
| give up, run my program and feel smug at my programming skills, if not my word- 
game skills. 


Conclusion 

Whoever said that “cheaters never win” didn’t take into consideration that cheating 
might lead to a better understanding of programming and data structures. And 
indeed, | often tell people that the key to good programming is often not knowing the 
best algorithms, but rather knowing which libraries to apply and how to combine the 
strengths of the language you're using, so that you can do as little work as possible. 
This article shows how to tackle a simple real-world problem that involved very little 
of your own code. Rather, understanding the problem, Python’s standard library and 
its data structures combined to give the answers. Ml 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email |jeditor@linuxjournal.com. 
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Fun with Mail 
Merge and Cool 


Dave Taylor has been hacking 
Basn Arravs iene 

systems for a really long time. 

He’s the author of Learning 


: : : : Unix for Mac OS X and 
Creating a sed-based file substitution tool. iced palsies 
You can find him on Twitter 
By Dave Taylor as @DaveTaylor, and you can 


reach him through his tech 
Q&A site Ask Dave Taylor. 
A few weeks ago, | was digging through my spam folder and 


found an email message that started out like this: 


Dear #name# 

Congratulations on winning the $15.7 million 
lottery payout! 

To learn how to claim your winnings, please... 


Obviously, it was a scam (does anyone actually fall for these?), 
but what captured my attention was the #name# sequence. 
Clearly that was a fail on the part of the sender who presumably 
didn’t know how to use AnnoyingSpamTool 1.3 or whatever the 
heck he or she was using. 


The more general notation for bulk email and file 
transformations is pretty interesting, however. There are 
plenty of legitimate reasons to use this sort of substitution, 
ranging from email newsletters (like the one | send every week 
from AskDaveTaylor.com—check it out!) to stockholder 
announcements and much more. 
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With that as the inspiration, let’s build a tool that offers just this capability 


The simple version will be a 1:1 substitution, so #name# becomes, say, “Rick Deckard”, 
while #f irst# might be “Rick” and #last# might be “Deckard”. Let’s build on that, 
but let’s start small. 


Simple Word Substitution in Linux 

There are plenty of ways to tackle the word substitution from the command line, 
ranging from Perl to awk, but here I’m using the original UNIX command sed (stream 
editor) designed for exactly this purpose. General notation for a substitution is s/old/ 
new/, and if you tack on ag at the end, it matches every occurrence ona line, not 
only the first, so the full command is s/old/new/g. 


Before going further, here’s a simple document that has necessary substitutions embedded: 


$ cat convertme.txt 
#date# 


Dear #name#, I wanted to start by again thanking you for your 
generous donation of #amount# in #month#. We couldn't do our 
work without support from humans like you, #first#. 


This year we're looking at some unexpected expenses, 
particularly in Sector 5, which encompasses #state#, as you 
know. I'm hoping you can start the year with an additional 
contribution? Even #suggested# would be tremendously helpful. 


Thanks for your ongoing support. With regards, 


Rick Deckard 
Society for the Prevention of Cruelty to Replicants 


Scan through it, and you'll see there’s a lot of substitutions to do: #date#, 
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#name#, #amount#, #month#, #first#, #state# and #suggested#. It turns 
out that #date# will be replaced with the current date, and #suggested# is one 
that'll be calculated as the letter is processed, but that’s for a bit later, so stay 
tuned for that. 


To make life easy, a source file that’s a comma-separated list allows for easy 
interaction with a source spreadsheet, so a sample input data file might look 
like this: 


name: first:amount:month: state 
Eldon Tyrell:Eldon:500:July:California 


At its most basic, the first line defines variable names (without the # notation), and 
subsequent lines are a set of values for a particular donor or recipient. To start, let’s 
read in the variable names: 


while IFS=',' read -r f1 f2 f3 f4 f5 f6 f7 
do 

declare -a varname=($f1 $f2 $f3 $f4 $f5 $f6 $f7) 
done 


Key to understanding this is to know about IFS, the internal field separator. 
Normally, it’s white space, which is why, for example, ls my file name looks 
for three files called my, file and name. But you can change it, as | demonstrate 
by changing IFS to a comma. 


Those Cool Bash Arrays 

| declare an array called varname that receives each of the fields read into the script. 
There are only five fields in use at this point, but let’s support up to seven to make the 
resultant script a bit more flexible. 


Arrays are really cool in Bash actually, but the notation is a smidge funky. That is, you 
can’t just use $array [index], because it won’t be parsed correctly, so curly braces 
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are a necessary addition: 
echo ${varname[1] } 
That works just fine. 


For a basic algorithm, you’re going to have two parallel arrays (parallel in that their 
indices will match up): one that retains all the variable names, and the other that 
contains the values for this instance of the data entry list. 


This means you'll need to differentiate between the situation when the script is 
reading the first line and when subsequent lines of the data file are read. Easily done 


(( Lines++ )) 


if [ $lines -eq 1] ; then # field names 
# variable names 
declare -a varname=($f1 $f2 $f3 $f4 $f5 $f6 $f7) 
else 
# values for this line (can contain spaces) 
declare -a value=("$f1" "$f2" "$f3" "$f4" "$f5" 
"Gio “Si? ) 
1 


As with most code, this makes assumptions here, but they’re safe: variable names 
aren't quoted because they’re always a single word, but variable values might 
have spaces, so they do end up quoted in the declare statement. Otherwise, this 
should be easy, and the (( Lines++ )) notation should make you cheer—it’s a 
nice Bash shortcut! 


Once you’re past the very first line, the script can look in varname[x] for the xth 
variable name, and value[x] for the value of that named variable, expressed as a 


series of sed-friendly substitution commands: 
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for ((i=0; i<${#value[*]}; i++)) 
do 
if [ ! -z "${value[$i]l}" ] ; then 
echo "s/#${varname [$i] }#/${value [$i] }/g" 
Ti 
done 


Which produces this: 


s/#name#/Eldon Tyrell/g 
s/#first#/Eldon/g 
s/#amount#/500/g 
s/#month#/July/g 
s/#state#/California/g 


That’s pretty darn close to what you want actually. Let’s push forward. 


Working with sed 
The stream editor sed is far more powerful than its modest and ancient history might 
suggest. It’s perfect for this job, as shown above. 


You could write the above lines into a temp file and invoke sed directly, but let’s avoid 
the file I/O and turn it all into a command-line argument as necessary. That’s done by 
simply separating each command with a semicolon, which you can do by building it in 
a temp variable instead: 


for ((i=0; i<${#value[x]}; i++)) 
do 
if [ ! -z "${value[$i]l}" ] ; then 
if [ -z "$SUBS" ] ; then 
SUBS="s/#${varname [$i] }#/${value [$i] }/g" 
else 
SUBS="$SUBS;s/#${varname [$i] }#/${value[$i] }/g" 
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Ti 
Ti 
done 


There’s undoubtedly a way to avoid the innermost if-then-else statement to omit the 
unnecessary ; prefix, but sometimes it’s easier to have a few lines of code than yet 
more gobbledygook. 


Otherwise, the above is a simple expansion from the previous for loop shown. This 
time, it builds the entire sed command within the SUBS substitution variable. Here’s 
how to test: 


echo " sed \"$SUBS\" $inputfile" 
When you run this with the input data file, here’s what’s pushed out to the terminal: 


sed "s/$name$/Eldon Tyrell/g;s/$first$/Eldon/g; 
s/$amount$/500/g;s/$month$/July/g; 
s/$state$/California/g" convertme.txt 

sed "s/$name$/Rachel/g;s/$first$/Rachel/g; 
s/$amount$/100/g;s/$month$/March/g; 
s/$state$/New York/g" convertme.txt 


(Note: line breaks added for formatting purposes only. ) 
It’s actually a very small step from here to invoke the command, so let’s do that: 


$ sub.sh 
#date# 


Dear Eldon Tyrell, I wanted to start by again thanking you 
for your generous donation of 500 in July. We couldn't do 


our work without support from humans like you, Eldon. 
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This year we're looking at some unexpected expenses, 
particularly in Sector 5, which encompasses California, as 
you know. I'm hoping you can start the year with an 
additional contribution? Even #suggested# would be 
tremendously helpful. 


Thanks for your ongoing support. With regards, 


Rick Deckard 
Society for the Prevention of Cruelty to Replicants 


$ 


Generally, this looks good. #date# and #suggested# are still untranslated, but that’s 
as expected. What is a bit odd is that it didn’t get the second entry too. A bug. 


’'m going to stop here, however, and maybe next time, I’ll add some system 
substitutions like #date# and figure out how to calculate #suggested#, which 
can be 50% of the actual donation. See you soon! mm 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email |jeditor@linuxjournal.com. 
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What’s New in Kernel Development 


By Zack Brown 


Chasing Archives 

Kernel development is truly impossible to keep track of. The 
main mailing list alone is vast beyond belief. Then there are all 
the side lists and IRC channels, not to mention all the corporate 
mailing lists dedicated to kernel development that never see 
the light of day. In some ways, kernel development has become 
fundamentally mysterious. 


Once in a while, some lunatic decides to try to reach back into 
the past and study as much of the corpus of kernel discussion 
as he or she can find. One such person is Joey Pabalinas, 
who recently wanted to gather everything together in Maildir 
format, so he could do searches, calculate statistics, generate 
pseudo-hacker Al bots and whatnot. 


He couldn't find any existing giant corpus, so he tried to create 
his own by piecing together mail archived on various sites. It 
turned out to be more than a million separate files, which was 
too much to host on either GitHub or GitLab. He asked the 
linux kernel mailing list for suggestions on better hosting 
opportunities. Although he acknowledged, “It’s possible I’m the 
only weirdo who finds this kind of thing useful, but | figured | 
should share it just in case I’m not.” 
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Joe Perches suggested plumbing the archives at kernel.org/lore.html, which 
go back decades. But Joey said he’d tried that, and he found it all but impossible 
to convert those archives to the Mailbox format he wanted. Instead, he’d spent 
the previous several weeks scraping the lkml.org archive and scripting his own 
conversion routines. 


Konstantin Ryabitsev remarked: 


The maildir format is kind of terrible for LKML, because having millions of 
messages in a single directory is very hard on the underlying FS. If you break it up 
into multiple folders, then it becomes difficult to search. This is the main reason 


why we have chosen to go with the public-inbox format, which solves both of 


these problems and allows for a very efficient archive updating and replication 


using git. 


Meanwhile, Jasper Spaans raised his eyebrows at Joey’s statement that hed gotten 
more than a million separate files by scraping Ikml.org. Jasper said: 


First of all, there are more than 3M messages stored in the Ikml.org database, so | 
guess you've missed some messages or something is really broken. Besides, unless 
you figured out how to get to the raw data, you’ve just scraped a rendering which 
discards stuff like pgp signatures etc and has very incomplete headers. Unless you 


don’t care for those of course. 


Jasper added that he’d also been working on extracting Maildir-type data out of the 
lore website, and he sent Joey the code he’d been using to do that. 


Eric Wong also sent Joey a script he’d been using to convert sirn threaded Usenet 
repositories to Maildir; although like others, he recommended against putting millions 


(and millions) of files into a single directory. 


The discussion wasn’t headed anywhere; it was just various people sharing knowledge 
and making judgment calls. 
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Once upon atime, anda very long time ago it was, | wanted to get a hold of the 
earliest archives of Linux kernel development discussions. | asked everyone where 

| could find them, and one of the developers replied that he had a lot of that stuff 
mixed up in his mail archives, along with all manner of other email messages. | 

wrote back and eagerly told him I'd love to get my hands on it. He wrote back again, 
explaining that there was just no way he could take the time to extract the private 
stuff from the public stuff. And, that was the end of that. I’ve always wondered why 
he responded to my initial email in the first place, if he was just going to say no at the 
end. And, that’s the tale of how | came this close to writing up summaries of the very 
earliest Linux developments. 


Considering Fresh C Extensions 

Matthew Wilcox recently realized there might be a value in depending on € 
extensions provided by the Plan 9 variant of the C programming language. All it 
would require is using the -fp Lan9-extensions command-line argument when 
compiling the kernel. As Matthew pointed out, Plan 9 extensions have been supported 
in GCC as of version 4.6, which is the minimum version supported by the kernel. So 
theoretically, there would be no conflict. 


Nick Desaulniers felt that any addition of -f compiler flags to any project always 


would need careful consideration. Depending on what the extensions are needed for, 
they could be either helpful or downright dangerous. 


In the current case, Matthew wanted to use the Plan 9 extensions to shave precious 
bytes off of a cyclic memory allocation that needed to store a reference to the “next” 


value. Using the extensions, Matthew said, he could embed the “next” value without 
breaking various existing function calls. 


Nick also suggested making any such extension dependencies optional, so that other 
compilers would continue to be able to compile the kernel. 


It looked as though there would be some back and forth on the right way to proceed, 
but Linus Torvalds immediately jumped in to veto the entire concept, saying: 
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Please don't. 


The subset of the plan9 extensions that are called just “ms” extensions is fine. That’s 
a reasonable thing, and is a very natural expansion of the unnamed structures we 


already have—namely being able to pre-declare that unnamed structure/union. 


But the full plan9 extensions are nasty, and makes it much too easy to write 
“convenient” code that is really hard to read as an outsider because of how the types 
are silently converted. 


And | think what you want is explicitly that silent conversion. 


So no. Don’t do it. Use a macro or an inline function that makes the conversion 


explicit so that it’s shown when grepping. 


The “one extra argument” is not a strong argument for something that simply 
isn’t that common. The upsides of a nonstandard feature like that needs to be 
pretty compelling. 


We've used various gcc extensions since day #1 (“inline” being perhaps the biggest 
one that took _forever_ to become standard C), but these things need to have very 


strong arguments. 


“One extra argument” that could be hidden by a macro or a helper inline is simply not 


a strong argument. 

Nick was sympathetic to this point, and said: 
Implicit conversions are the most pointed to “defect” in languages like JavaScript. | 
understand why they’re convenient, but the resulting surprising bugs don’t outweigh 


their cost (again, my opinion), and developers frequently can’t keep these implicit 
conversion rules straight (whether we’re talking about JavaScript or C). 
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There was no real conversation after Linus’ denunciation. But, the question 

of which compiler extensions to use and which not to use is very interesting, 
especially for a project like Linux that’s used everywhere and runs everything. 

With all the different environments it needs to support, that support needs to 
include the possibility of all sorts of development platforms. To some extent, Linus’ 
argument that Matthew was trying to create hard-to-read code is relevant, but it’s 
also relevant that other compilers and other build environments also need to be 
able to support Linux development. 


Handling Complex Memory Situations 

Jérome Glisse felt that the time had come for the Linux kernel to address 

seriously the issue of having many different types of memory installed on a single 
running system. There was main system memory and device-specific memory, and 
associated hierarchies regarding which memory to use at which time and under which 
circumstances. This complicated new situation, Jéréme said, was actually now the 
norm, and it should be treated as such. 


The physical connections between the various CPUs and devices and RAM chips—that 
is, the bus topology—also was relevant, because it could influence the various speeds 
of each of those components. 


Jéréme wanted to be clear that his proposal went beyond existing efforts to handle 
heterogeneous RAM. He wanted to take account of the wide range of hardware and 
its topological relationships to eek out the absolute highest performance from a given 
system. He said: 


One of the reasons for radical change is the advance of accelerator like GPU or 

FPGA means that CPU is no longer the only piece where computation happens. It 

is becoming more and more common for an application to use a mix and match of 
different accelerator to perform its computation. So we can no longer satisfy our self 
with a CPU centric and flat view of a system like NUMA and NUMA distance. 


He posted some patches to accomplish several different things. First, he wanted to 
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expose the bus topology and memory variety to userspace as a clear API, so that both 
the kernel and user applications could make the best possible use of the particular 
hardware configuration on a given system. A part of this, he said, would have to 

take account of the fact that not all memory on the system always would be equally 
available to all devices, CPUs or users. 


To accomplish all this, his patches first identified four basic elements that could be 
used to construct an arbitrarily complex graph of CPU, memory and bus topology on 
a given system. 


These included “targets”, which were any sort of memory; “initiators”, which were 
CPUs or any other device that might access memory; “links”, which were any sort of 
bus-type connection between a target and an initiator; and “bridges”, which could 
connect groups of initiators to remote targets. 


Aspects like bandwidth and latency would be associated with their relevant links and 
bridges. And, the whole graph of the system would be exposed to userspace via files 
in the SysFS hierarchy. 


In addition, Jéréme’s patches provided a way to express memory policy. A system’s 
memory policy is the mechanism it uses to decide which memory to use for a given 
task. For example, it might use faster memory first and slower memory only when fast 
memory is full. But, the kernels current memory policy was organized on a per-CPU 
basis, which Jéréme felt was not good enough. But, he also acknowledged that trying 
to change that aspect of kernel infrastructure directly might break a lot of existing 
code. To deal with this, his patch added an entirely new memory policy API that new 
user code could take advantage of and old user code simply could ignore. 


Aneesh Kumar responded to all of this, in particular praising JéroOme’s approach 
of keeping the new API separate from the old. But Aneesh said, “that is also the 
drawback isn’t it? We now have multiple entities tracking cpu and memory.” 


Aneesh also wanted to confirm that “once we have these different types of targets, 


73 | March 2019 | https://www.linuxjournal.com 


diff -u 


ideally the system should be able to place them in the ideal location based on the 
affinity of the access. ie. we should automatically place the memory such that initiator 
can access the target optimally.” 


Jéréme seemed to agree with this in principle, but he also seemed to feel that making 
any of this automatic was still not guaranteed. The first step, he felt, was to expose 


the APIs and data structures, and then see what could be accomplished. 


Meanwhile, Dave Hansen pointed out that there were existing elements of the 


kernel that dealt with heterogeneous memory. Dave said that HMAT (Heterogeneous 
Memory Attribute Table) existed in firmware specifically to express the topology to 
the kernel. Dave also said that NUMA (Non-Uniform Memory Access) was already 


part of the kernel, and wasn’t lying fallow. Additionally, he pointed out that the ACPI 
(Advanced Configuration and Power Interface) specification had officially embraced 
NUMA, and there were Linux developers actively contributing patches to support this. 


So, Dave was not immediately enthusiastic about ditching this ongoing momentum 
in one direction, in order to accept Jérdme’s radical solution that went in an entirely 
new direction. 


But, Jéroéme replied that he was not trying to overthrow the existing work or any 
kernel patches that made use of HMAT. He said that all of that was still useful just on 
its own. But he added: 


| do not see how to evolve NUMA to support device memory. [...] | can not expose 
device memory as NUMA node as device memory is not cache coherent on AMD and 
Intel platform today. [...] In some case that memory is not visible at all by the CPU 
which is not something you can express in the current NUMA node. 


Somewhat mollified, Dave replied: 


Yeah, our NUMA mechanisms are for managing memory that the kernel itself manages 
in the “normal” allocator and supports a full feature set on. That has a bunch of 
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implications, like that the memory is cache coherent and accessible from everywhere. 


The HMAT patches only comprehend this “normal” memory, which is why we’re 


extending the existing /sys/devices/system/node infrastructure. 


This series has amuch more aggressive goal, which is comprehending the connections 


of every memory-target to every memory-initiator, no matter who is managing the 


memory, who can access it, or what it can be used for. 


Theoretically, HMS could be used for everything that we’re doing with /sys/devices/ 
system/node, as long as it’s tied back into the existing NUMA infrastructure somehow. 


Jéréme agreed with all of the above, and Dave seemed to be on board with Jéréme’s 
approach. But, he did have some practical objections. For one thing, he said: 


We support 1024 NUMA nodes on x86. The ACPI HMAT expresses the connections 
between each node. Let’s suppose that each node has some CPUs and some memory. 


That means we'll have 1024 target directories in sysfs, 1024 initiator directories 
in sysfs, and 1024* 1024 link directories. Or, would the kernel be responsible for 
“compiling” the firmware-provided information down into a more manageable 


number of links? 


Some idiot made the mistake of having one sysfs directory per 128MB of memory 
way back when, and now we have hundreds of thousands of /sys/devices/system/ 


memory/memoryX directories. That sucks to manage. Isn’t this potentially 


repeating that mistake? 


Dave also was worried that if JéroOme went forward with his patches, it could be four 
or five years before all the problems were solved, in which case, some portions of 
memory management would be bottle-necked waiting for those solutions, which 
could have been solved sooner using existing NUMA projects. 
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Additionally, Dave was curious how Jéréme’s code would scale. He said, “It’s quite 
easy to represent a laptop, but can this scale to the largest systems that we expect to 
encounter over the next 20 years that this ABI will livee” 


At this point, JérOme and Dave were joined by several other folks and began diving 
into the technical details, further objections and possible solutions that might come 
out of Jéréme’s work. Ultimately, it seemed as if these patches did not represent a 
threat to existing approaches to memory, and that Jér6me would have support—or at 
least tolerance—from NUMA-related projects. 


The things | love about this discussion are, first of all, that one developer got a big 
idea that seemed to go against current thinking, but that solved problems he saw as 
real. Second, that a developer on the other side of the issue was actually interested in 
the new approach and willing to take it seriously rather than tear it down. 


Also, the whole direction of hardware resources is really becoming so strange. The 
kernel tries to eek out absolutely everything it can from the various devices on the 
system—even to the point of going beyond the ways in which those devices thought 
they would be used! And then once the kernel starts using them that way, other 
devices come out that use the kernel’s new infrastructure. And so we end up with 
some kind of crazy situation requiring crazy solutions like what Jéréme has proposed. 


Note: if you’re mentioned in this article and want to send a response, please send 
a message with your response text to Ijeditor@linuxjournal.com and we’ll run 

it in the next Letters section and post it on the website as an addendum to the 
original article. ™ 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email |jeditor@linuxjournal.com. 
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Arduino from the 
Command Line: 
Break Free from the 
GUI with Git and Vim! 
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Love Arduino but hate the GUI? Try arduino-cli. 
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In this article, | explore a new tool released by the Arduino team that can free you 
from the existing Java-based Arduino graphical user interface. This allows developers 
to use their preferred tools and workflow. And perhaps more important, it'll enable 
easier and deeper innovation into the Arduino toolchain itself. 


The Good-Old Days 


When | started building hobby electronics projects with microprocessors in the 
1990s, the process entailed a discrete processor, RAM, ROM and masses of glue logic 
chips connected together using a point-to-point or “wire wrapping” technique. (Look 
it up kids!) Programs were stored on glass-windowed EPROM chips that needed 


Figure 1. 
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Figure 2. 
Example Mid-1990s 
Microprocessor 
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to be erased under UV light. All the tools were expensive and difficult to use, and 
development cycles were very slow. Figures 1-3 show some examples of my mid- 
1990s microprocessor projects with discrete CPU, RAM and ROM. Note: no Flash, no 
0, no DACs, no ADCs, no timers—all that means more chips! 


It all changed in 2003 with Arduino. 


The word “Arduino” often invokes a wide range of opinions and sometimes emotion. 
For many, it represents a very low bar to entry into the world of microcontrollers. 
This world before 2003 often required costly, obscure and closed-source 
development tools. Arduino has been a great equalizer, blowing the doors off the 
walled garden. Arduino now represents a huge ecosystem of hardware that speaks 

a (mostly) common language and eases transition from one hardware platform to 
another. Today, if you are a company that sells microcontrollers, it’s in your best 
interest to get your dev boards working with Arduino. It offers a low-friction path to 
getting your products into lots of hands quickly. 
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Example Mid-1990s 
Microprocessor 


It’s also important to note that Arduino’s simplicity does not inhibit digging deep into 
the microcontroller. Nothing stops you from directly twiddling registers and using 
advanced features. It does, however, decrease your portability between boards. 


For people new to embedded development, the Arduino software is exceptionally 
easy to get running. That has been a major reason for its success—it dumps out of 
the box ready to rock. But for more seasoned developers, the included graphical user 
interface can be frustrating. It can be a barrier to using modern development tools 
and version control like Git. | know the compiler and toolchain is buried deep in that 
GUI somewhere. | just want to use my favorite editor, compile my code and upload 
my project to a dev board using my favored workflow. For many developers, this is a 
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command-line or scripted process. 


Enter arduino-cli 

There have been a couple attempts to break out Arduino to the command line, but 
most failed to get wide support. However, now the Arduino team has alpha-released 
arduino-cli. This new framework not only provides a comprehensive set of command- 
line functions, but the team also says it will be used as the core underneath the 

next generation of the Arduino graphical interface. This is exciting news and shows 
commitment to this new concept. 


For me, my preferred development workflow is using Git for version control and the 
Vim editor, so that’s what | demonstrate in the remainder of this article. 


Installing arduino-cli 

At the time of this writing, the arduino-cli is in alpha release. It’s being distributed 
both as a Go source package and pre-built binaries. The Go language produces very 
portable binaries, so you just need to download the correct file and put the binary 
somewhere in your $PATH. Most users reading this likely will want the Linux 64-bit for 


Intel/AMD systems. In the examples here, my system happens to be running Fedora 
29, but any recent Linux should work. Check the project’s GitHub page for updated 


versions; at the time of this writing, 0.3.2 is the latest alpha release. Finally, make 
sure your user can access serial- and USB-connected Arduino devboards by adding 
them to the “dialout” group (note: you'll need to re-log in to pick up the new group 
membership, and substitute “me” for your user name in the last command): 


me@mybox:~ $ wget https://downloads.arduino.cc/arduino-cli/ 
sarduino—cli-Q.3.2-alpha.preview-linux64. tar.bz2 
**kdOwN Load ing*** 

me@mybox:~ $ tar -xjf arduino-cli-@.3.2-alpha.preview- 
olinux64.tar.bz2 

me@mybox:~ $ sudo mv arduino-cli-@.3.2-alpha.preview-Linux64 
o/usr/local/bin/arduino-cli 

me@mybox:~ $ sudo usermod —a -G dialout me 
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Assuming /usr/local/bin is in your $PATH, you should be able to run arduino-cli as any 
user on your Linux system. 


Alternatively, if you want to build the Go package from source, you can use the 
get function to download, build and install the source package from the Arduino 
GitHub repository 


me@mybox:~ $ sudo dnf -y install golang 

me@mybox:~ $ cd ; export GOPATH='pwd'/go 

me@mybox:~ $ go get -u github.com/arduino/arduino-cli 
me@mybox:~ $ sudo mv $GOPATH/bin/arduino-cli /usr/local/bin/ 


Arduino from the Command Line 

First, let’s do some housekeeping. You need to pull over the current index of Arduino 
“cores” and search for the core that supports your dev board. In this first example, 
let’s install support for the classic UNO board powered by an ATMega AVR processor: 


me@mybox:~ $ arduino-cli core update-index 
Updating index: package_index.json downloaded 
me@mybox:~ $ arduino-cli core search avr 
Searching for platforms matching ‘avr' 


ID Version Name 

arduino:avr 1s 6623 Arduino AVR Boards 
arduino:megaavr 1.6.24 Arduino megaAVR Boards 
atmel-avr-xminis:avr 0.6.0 Atmel AVR Xplained-minis 
emoro:avr 34202 EMORO 2560 

LittleBits:avr 13040 LittleBits Arduino AVR Modules 


me@mybox:~ $ arduino-cli core install arduino:avr 
**kk Lots of downloading omitted **x 

me@mybox:~ $ arduino-cli core list 

ID Installed Latest Name 
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arduino:avr@1.6.23 1.0.25 1.6.23 Arduino AVR Boards 


That’s it. You have everything you need to create a new Arduino project for an 
Arduino UNO board. Now, let’s create a project called myBlinky. You'll also initialize 
and set up a Git repository to manage version control, then make your first commit: 


me@mybox:~ $ arduino-cli sketch new myBlinky 

Sketch created in: /home/me/Arduino/myB Linky 
me@mybox:~ $ cd /home/me/Arduino/myBlinky 
me@mybox:~/Arduino/myBlinky $ git config --global 
ouser.email '"me@mybox. com" 
me@mybox:~/Arduino/myBlinky $ git config --global 
ouser.name "My Name" 

me@mybox:~/Arduino/myBlinky $ git init 

Initialized empty Git repository in /home/me/Arduino/ 
omyBlinky/.git/ 

me@mybox:~/Arduino/myBlinky $ ls -la 

total 16 
drwxr-xr-x me me 4096 Nov 22 10:45 . 

me me 4096 Nov 22 10:45 .. 

me me 4096 Nov 22 10:45 .git 

me me 35 Nov 22 10:45 myBlinky. ino 


drwxr-xr-x 
drwxr-xr-x 


PN W W 


—rw-r--r-- 


me@mybox:~/Arduino/myBlinky $ cat myBlinky.ino 


void setup() { 
} 


void loop() { 
, 


me@mybox:~/Arduino/myBlinky $ git add -A 
me@mybox:~/Arduino/myBlinky $ git commit -m "Initial Commit" 
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[master (root-commit) ee95972] Initial Commit 
1 file changed, 6 insertions(+) 
create mode 100644 myBlinky.ino 


me@mybox:~/Arduino/myBlinky $ git log 

commit ee9597269c5da49d573d6662 fe8f8137083b7768 
«(HEAD -> master) 

Author: My Name <me@mybox. com> 

Date: Thu Nov 22 10:48:33 2018 -@500 


Initial Commit 


Nice! The tool creates the project under the same Arduino directory structure 
where the graphical tools would expect to find them allowing you to flip between 
tools if you wish. It also creates a template .ino file with the familiar setup () 
and loop() functions. 


The empty Git repository was initialized, and you can see it created the .git 
subdirectory where all the version data will be kept. Time to code! 


Now, simply open up myBlinky.ino in your preferred editor—which in the interest of 
maximum street cred is Vim of course. Never EMACS (joking!)...seriously, use any 
editor you like (| hear Eclipse is nice) —then type in and save a classic “hello world” 


blinky program. Something like this: 


// Simple Demo Blinker -—MEH 
#define PIN_LED 13 


void setup() { 
pinMode(PIN_LED, OUTPUT); 


void loop() { 


86 | March 2019 | https://www.linuxjournal.com 


DEEP 
DIVE 


digitalWrite(PIN_LED,HIGH) ; 
delay(500) ; 
digitalWrite(PIN_LED,LOw) ; 
delay(500); 


Now, let’s compile and upload it to the Arduino UNO. Use the board set of 
commands to search for the upload: 


me@mybox:~/Arduino/myBlinky $ arduino-cli board list 
FQBN Port ID Board Name 
arduino:avr:uno /dev/ttyACMQ 2341:0001 Arduino/Genuino Uno 


It found the board. Now compile: 


me@mybox:~/Arduino/myBlinky $ arduino-cli compile —-b 
earduino:avr:uno 

Build options changed, rebuilding all 

Sketch uses 930 bytes (2%) of program storage space. Maximum 
461s 32256 bytes. 

Global variables use 9 bytes (0%) of dynamic memory, leaving 
+2039 bytes for local variables. Maximum is 2048 bytes. 


Compile was clean. Next, upload to the board using the upload command, board 
name and port you discovered earlier: 


me@mybox:~/Arduino/myBlinky $ arduino-cli upload 
o-b arduino:avr:uno -p /dev/ttyACMQ@ 


As is common with command-line tools, silence is golden. The upload command 


completes, and the UNO is happily blinking away. You better lock in this good fortune 
with agit commit: 


87 | March 2019 | https://www.linuxjournal.com 


DEEP 
DIVE 


me@mybox:~/Arduino/myBlinky $ git commit -a -m "It works! 
oFirst blink." 

[master 35309a0] It works! First blink. 

1 file changed, 7 insertions(+) 


The -m option takes a commit message; it should be a note about what’s included 
in this commit. If you omit the message, git will open a template message in a text 
editor (the default is Vim, but you can change it by setting $EDITOR). 


Support for Third-Party Development Boards 

Now for something a little more ambitious, let’s try to use a non-Arduino 
development board and walk through the steps of adding a third-party core. This can 
be tricky even with the graphical user interface, but it’s pretty simple with arduino-cli. 
Let’s target the very popular ESP8266. 


For starters, you need to add the third-party repository to a file so arduino-cli knows 
how to locate the core. This is done with a file named .cli-config.yml. You might think 
this should go in your home directory or in the project directory, and you would be 
right to think that. But, one quirk of this early release of arduino-cli is that this file 
lands in the same directory where the arduino-cli program lives. For now, this is /usr/ 
local/bin, but keep an eye on the website, as this is likely to change in future releases! 
Below, you'll add a new board config definition. This file uses YAML format, so be 
careful to use only spaces in the indenting. Edit (with sudo), and place the following 
text in /usr/local/bin/.cli-config.yml: 


board_manager: 
additional_urls: 
— http://arduino.esp8266.com/stable/ 
opackage_esp8266com_index. json 
Now, like before, update the index and install the core: 


me@mybox:~/Arduino/myBlinky $ arduino-cli core update-index 


88 | March 2019 | https://www.linuxjournal.com 


DEEP 
DIVE 


Updating index: package_index.json downloaded 
Updating index: package_esp8266com_index.json downloaded 


You can see that it found and downloaded the index for esp8266 cores. Good. Now, 
let's download and install the core itself: 


me@mybox:~/Arduino/myBlinky $ arduino-cli core search esp 
Searching for platforms matching ‘esp' 
ID Version Name 
esp8266:esp8266 2.4.2 esp8266 


me@mybox:~/Arduino/myBlinky $ arduino-cli core install 
6e€S5p8266: esp8266 

Downloading esp8266:esptool@@.4.13... 
esp8266:esptool@d.4.13 downloaded *x* much omitted **x 


Now, you can rebuild your myBlinky project for the esp8266 and upload it. You first 
need to edit your myBlinky.ino and change the #define PIN_LED to whichever pin 
has the LED. On my dev board, it’s pin 2. Make that modification and save it: 


#define PIN_LED 2 


After plugging in the esp8266 dev board, you again run the board list command to 
try to find it: 


me@mybox:~/Arduino/myBlinky $ arduino-cli board list 
FQBN Port ID Board Name 
/dev/ttyUSBO 1a86: 7523 unknown 


It detects it, but it can’t determine what it is. That is quite common for boards like the 
esp8266 that require a button push or special programming mode. 
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Next, compile and upload after resetting your esp8266 board while pressing the 
program button. Like last time, use the board name and port discovered during the 
list operation: 


me@mybox:~/Arduino/myBlinky $ arduino-cli board listall 

o|grep esp8266 

Generic ESP8266 Module esp8266:esp8266: generic 
(omitted long list) 

me@mybox:~/Arduino/myBlinky $ arduino-cli upload -b 
+6eSp8266:esp8266:generic -—p /dev/ttyUSBQO 

Uploading 252000 bytes from /home/me/Arduino/myBlinky/ 
omyBlinky.esp8266.esp8266.generic.bin to flash at 0x00000000 


iivadais [ 100% ] 
And, it blinks! Make another git commit and save your progress 


me@mybox:~/Arduino/myBlinky $ git add -A 
me@mybox:~/Arduino/myBlinky $ git commit -m 

o"Blinking on esp8266 board" 

[master 2ccffid] Blinking on esp8266 board 

5 files changed, 61 insertions(+), 1 deletion(-) 

create mode 100755 myBlinky.arduino.avr.uno.elf 

create mode 100644 myBlinky.arduino.avr.uno.hex 

create mode 100644 myBlinky.esp8266.esp8266.generic.bin 
create mode 100755 myBlinky.esp8266.esp8266.generic.elf 


As you can see, it saves the compiled binary versions of the compiled code in the 
project directory. You can add *.bin, *.hex and *.elf to a .gitignore file if you wish to 
omit these from your commits. IF you do save them, you can use the -i option and 
the .bin file to upload a specific binary. 
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Adding Libraries 


Building on your success, you should download and install a library. Let’s up the blinky 
game and install support for some Adafruit NeoPixels (aka WS2812B, PL9823 and so 


on). First, search for it using the Lib command, then download and install: 
me@mybox:~/Arduino/myBlinky $ arduino-cli lib search neopixel 
(omitting large list) 

me@mybox:~/Arduino/myBlinky $ arduino-cli lib install 
o"Adafruit NeoPixel" 

Adafruit NeoPixel@1.1.7 downloaded 

Installed Adafruit NeoPixel@1.1.7 

Now you need to modify the program; edit the .ino file with these modifications: 


// Fancy NeoPixel Blinky Blinker 


#include <Adafruit_NeoPixel.h> 
#define PIN_LED 14 


Adafruit_NeoPixel strip = Adafruit_NeoPixel(1, PIN_LED, 
oNEO_GRB + NEO_KHZ80Q) ; 


void setup() { 
strip.begin(); 

void loop() { 
strip.setPixelColor(Q,strip.Color(255,0,0)) ' 
delay(200); 
strip.setPixelColor(0,strip.Color(0,255,0)); 


delay(2Q00); 
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strip.setPixelColor(0,strip.Color(0,0,255)); 
delay(200); 


Next, do the same compile and upload dance, after, of course, attaching a NeoPixel or 
compatible LED to pin 14: 


me@mybox:~/Arduino/myBlinky $ arduino-cli compile —-b 
+eSp8266:eSp8266: generic 

Build options changed, rebuilding all 

Sketch uses 248592 bytes (49%) of program storage space. 
oMaximum is 499696 bytes. 
Global variables use 28008 bytes (34%) of dynamic memory, 
o6leaving 53912 bytes for local variables. Maximum 

ois 81920 bytes. 

me@mybox:~/Arduino/myBlinky $ arduino-cli upload -b 
4eSp8266:esp8266:generic -p /dev/ttyUSBO 

Uploading 252960 bytes from /home/me/Arduino/myBlinky/ 
omyBlinky.esp8266.esp8266.generic.bin to flash at 0x00000000 


pave ne wien [ 100% ] 


And, you should have a colorful blinky—pretty slick. It’s another good time to commit 
your changes to capture your progress: 


me@mybox:~/Arduino/myBlinky $ git add -A 
me@mybox:~/Arduino/myBlinky $ git commit -m 
o"Blinky with NeoPixels" 

[master 122911f] Blinky with NeoPixels 

3 files changed, 20 insertions(+), 13 deletions(-) 
rewrite myBlinky.ino (81%) 
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Using GitHub 

Up to now, the git repository has been completely local in your project directory. 
There’s nothing wrong with that, but let’s say you want to publish your work to 
GitHub. It’s pretty quick and easy to get started. First, log in to github.com and 
create an account if you don’t already have one. Then, click the button to create 
a “new repository”. 


Fill in the details for your project, and be sure to leave unchecked the option 
to initialize the repository with a README. This is because you already have a 


Figure 4. 
Repositories New repository New Repository 


Figure 5. 
Create Repo 


' Create a new repository 


A repository contains all the files for your project, including the revision history. 


Owner Repository name 
pa sysmatt~ /  myBlinky S 
Great repository names are short and memorable. Need inspiration? How about supreme-umbrella. 


| Description (optional) 
| | Aplace to commit all my blinkyness!| 


i 
| @ [| Public 
/ “#— Anyone can see this repository. You choose who can commit. 


\ 
'@ Private 
\ 

You choose who can see and commit to this repository. 


| (©) Initialize this repository with a README 
This will let you immediately clone the repository to your computer. Skip this step if you're importing an existing repository. 


Add .gitignore: None ~ Addalicense:Nonex @) 


Create repository 
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git remote add origin https://github.com/sysmatt/myBlinky.git Push Existing 


git push -u origin master 


Repository 


repository created, and you'll be “pushing” your local repository to GitHub, so 
you want it empty. 


After creating it, you will be presented with some helpful options on how to push 
code into GitHub. You want the commands for “push existing repository”. 


Now let’s do it! Follow the instructions to create a git “remote” entry named 
“origin”, which will represent GitHub. Then you will push, and it will prompt you 
for your GitHub user name and password (substitute your own GitHub URL, user 
name and password): 


me@mybox:~/Arduino/myBlinky $ git remote add origin 
ohttps://github.com/sysmatt/myBlinky.git 
me@mybox:~/Arduino/myBlinky $ git push -u origin master 
Username for ‘'https://github.com': sysmatt 

Password for ‘'https://sysmatt@github. com’: seb 
Counting objects: 18, done. 

Delta compression using up to 4 threads. 

Compressing objects: 100% (15/15), done. 

Writing objects: 100% (18/18), 825.31 KiB | 5.00 MiB/s, done. 
Total 18 (delta 4), reused @ (delta Q) 

remote: Resolving deltas: 100% (4/4), done. 

* [new branch] master -> master 

Branch master set up to track remote branch master from origin. 


Now, if you browse to the repository on GitHub.com, you'll see all your files 
and every one of your commits. You can travel back in time and look at every 
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© sysmatt / myBlinky ©unwatchy 1 wWStar 0 York 0 Figu re i. 
<> Code Issues 0 Pull requests 0 Projects 0 Wiki ii Insights Settings Commit H istory 


Branch: master + 


Commits on Nov 22, 2018 


Blinky with NeoPixels f tazotit © 


My Name committed 31 minutes ago 


Blinking on esp8266 board GB 2ecffid © 


My Name commit tted 10 hours ago 


It works! First blink. fe 353090 © 


My Name commit tted 11 hours ago 


Initial Commit fe ee95972 © 


My Name commit tted 11 hours ago 


revision. In practice, you can use SSH keys to eliminate having to enter your 
password every time. 


Time Travel (aka Breaking the Laws of Physics 
with Git) 

Time travel, you say? Yes indeed. Let’s say you want to jump back in time and 
review your version of myBlinky from when you had it working with the Arduino 
UNO. It seems like ages ago. It’s easy! You just need to identify the commit id and 
“checkout” that version. 


Use the Log command to list all your commits: 


me@mybox:~/Arduino/myBlinky $ git log 

commit 122911f99dddce2dabbb251c3b640c8c7f9F98d9 (HEAD -> 
omaster, origin/master) 

Author: My Name <me@mybox. com> 

Date: Thu Nov 22 21:22:59 2018 -@500 


Blinky with NeoPixels 


commit 2ccff1d7326b1523649510f24644c96df6dc6e12 
Author: My Name <me@mybox. com> 


95 | March 2019 | https://www.linuxjournal.com 


DEEP 
DIVE 


Date: Thu Nov 22 11:42:02 2018 -0500 


Blinking on esp8266 board 
commit 35309a0c9e90668052abc9644f77F906ab57949c 
Author: My Name <me@mybox. com> 
Date: Thu Nov 22 11:09:44 2018 -@500 


It works! First blink. 


commit ee9597269c5da49d573d6662 fe8f8137083b7768 
Author: My Name <me@mybox. com> 
Date: Thu Nov 22 10:48:33 2018 -0500 


Initial Commit 


It looks like the commit starting with 35309a0c is the one youre after. Note: you can 
shorten the commit hash string to as few as four characters, as long as it uniquely 
identifies only one commit. Let’s explore that version: 


me@mybox:~/Arduino/myBlinky $ git checkout 35309aQc 
HEAD is now at 35309a@... It works! First blink. 
me@mybox:~/Arduino/myBlinky $ ls -l 

—rw-r--r--— 1 me me 191 Nov 22 22:01 myBlinky. ino 
me@mybox:~/Arduino/myBlinky $ cat myBlinky.ino 

// Simple Demo Blinker -—MEH 

#define PIN_LED 13 


void setup() { 


pinMode(PIN_LED, OUTPUT) ; 


void loop() { 
digitalWrite(PIN_LED,HIGH); 
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delay(500) ; 
digitalWrite(PIN_LED,LOw) ; 
delay(500); 


Now let’s say you’re done poking around, so let’s time travel forward to the 
current day and get things back to before you broke the laws of physics. In the 
simple git repository, this means jumping back to the current commit in the 
“master” branch: 


me@mybox:~/Arduino/myBlinky $ git checkout master 
Previous HEAD position was 35309a@... It works! First blink. 
Switched to branch ‘master’ 

Your branch is up-to-date with ‘origin/master'. 

me@mybox:~/Arduino/myBlinky $ ls -l 

total 1332 

=rwxr=—xr—-x 1 me me 13956 Nov 22 22:04 
omyBlinky.arduino.avr.uno.elf 

=rw-r=—r=—— 1 me me 2640 Nov 22 22:04 
omyBlinky.arduino.avr.uno.hex 

—-rw-r--r-- 1 me me 252960 Nov 22 22:04 
omyBlinky.esp8266.esp8266.generic.bin 

—rwxr-xr-x 1 me me 1082905 Nov 22 22:04 
omyBlinky.esp8266.esp8266.generic.elf 

=Ffw-r--=r=— 1 me me 436 Nov 22 22:04 myBlinky.ino 


Nice! You’re back to the NeoPixel blinky. You see, git stores all the versions 

of your committed code under the .git subdirectory. This is actually the real 
location of the repository. The files you edit are just the “work area”. When you 
jump around between commits, git is doing all the magic of modifying the files in 
the work area. If you wanted to jump back and start modifying the old version of 
code, you could create a new “branch” to contain that work and move forward 
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with an AVR and esp8266 fork of your code. It’s very powerful. 


’ve barely scratched the surface here. Git, GitHub and arduino-cli are all quite 
comprehensive tools. | hope this article has given you a taste of what’s possible when 
you harness good programming workflows for your Arduino projects. m 


Matthew Hoskins is the Senior Enterprise Architect at New Jersey Institute of Technology where he leads the team of talented 
professionals that keep our dizzying array of services working together. Systems, storage, on-premises and in the cloud, we do it all. 
When it’s done right, no one notices. Matt is @SYSMATT on Twitter. 


Resources 


¢ Arduino-CL! Announcement 


¢ Arduino-CLl GitHub 


Send comments or feedback 
via https://www.linuxjournal.com/contact 
or email |jeditor@linuxjournal.com. 
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Indie Makers 
Using Single- 
Board Computers 


Possibly the most amazing thing, to me, about single-board 
computers (SBCs) is that they allow small teams of people (and 
even lone individuals) to create new gadgets using not much more 
than SBCs and 2D printers. That opportunity for makers and small 
companies is absolutely astounding. 


By Bryan Lunduke 


Two such projects have really caught my attention lately: the Noodle Pi and the TinyPi. 


The Noodle Pi is asimple, handheld computer (about the size of a deck of playing 
cards). And, when | say simple, | mean simple. It’s got a micro-USB charging port, 
another for plugging in USB devices, a touch screen and a battery. Think of it like an 
old-school PDA without any buttons (other than a small power toggle) and the ability 
to run a full Linux-based desktop. 


The TinyPi is a gaming handheld. And, believe it or not, it’s even smaller than the 
Noodle Pi, with a tiny screen and tiny buttons. This is the sort of handheld game 
console you could put on a keychain. 


Both of these are built on top of the (super-tiny and super-cheap) Raspberry Pi Zero. 
And, both are built by lone individuals with a heavy reliance on 3D printers. 
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Figure 1. The Credit-Card-Sized, Pi Zero-Powered, Noodle Pi 


Figure 2. 

The Impossibly 
Small TinyPi 
(Banana for Scale) 
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| wanted to know how they did it and how their experience was. What can we learn 
from these independent gadget makers? So, | reached out to both of them and asked 
them each the same questions (more or less). 


Let’s start with a chat with Pete Barker (aka “piOcket”) , maker of the TinyPi. 


Interview with Pete Barker (piOcket), TinyPi Maker 


Bryan Lunduke: Could you give a quick overview of the TinyPic 


Pete Barker: TinyPi is (unofficially) the world’s smallest pi-based gaming device. It 
started life as a bit of a joke—”how small can i make thise’—but it actually turned into 
something pretty good. The Pro version added more features and improvements, and 
a kickstarter was funded on December 30, 2018. Manufacturing is already underway, 


and the early-bird backers should start getting the kits in February 2019. 


BL: When you first started working on the TinyPi, was it always envisioned to be 
based on a Raspberry Pi Zero? 


PB: It actually was the release of the Zero that sparked the project. | already had been 
playing with the Raspberry Pi, but the reduced footprint was a game-changer. 


BL: Any other boards that were considered back then? 


PB: At the time, the zero was the best out there for the price and size. There were 
some “banana” and “orange” pis but they tended to be alittle on the large size. 


BL: Did the Pi Zero meet your needs? Was it...fast enough? 

PB: We always want more speed. The Zero would be great if it had the power of 
a Pi3, but then it would eat the batteries. The Zero does a great job of emulating 
retrogames, and it also works well for other things. 


BL: If you had it to do all over again, what sort of single-board computer would you use? 
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TinyPiPro 


Figure 3. The parts of the TinyPi—the Smallest Handheld Game Console | Can Possibly Imagine 


PB: | have been investigating alternatives. There are a few “NanoPi” boards that have 
quad-core processors in small footprints. There is also the BeagleBone on-chip, which 
lets you build your own SBC with very few components. The biggest draw to the 
Raspberry Pi boards is the software support. Because the RPis are super popular and 
well established, there’s a fast range of software ready to roll. 


BL: What part of building something around a single-board computer was the least 
enjoyablee Any headaches? 


PB: There were no real headaches. It’s all been a learning curve throughout the 
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project. Two years ago, | had never designed a PCB or used any form of CAD. 


BL: Do you use Pis (or other similar boards) in your own life, outside the TinyPi? 


PB: | actually have a scary number of Raspberry Pis in my house. One is running DNS 
filtering and freeview decoding, and one is on each TV playing media from a NAS 
box. | have multiple handhelds filled with Pi, and ’m even working on a pocket-sized 
keyboard and screen combo called “clicker”. 


BL: If you could change one thing about the Pi Zero, what would it be? 


PB: A Pi Zero with the power of the Pi3 would be great, but | don’t think that will 
become a reality. And a smaller footprint with maybe castellated edges would mean 
that | could make something even smaller (something like this, but not $99). 


And now let’s pose those same questions to Ashish Gulhati, maker of the Noodle Pi. 


Interview with Ashish Gulhati, Noodle Pi Maker 


Bryan Lunduke: What is the Noodle Pi? 


AG: Okay, so the Noodle Pi is a handheld PC based on the Raspberry Pi Zero, witha 
hi-res touchscreen, and integrated camera and battery. End users can assemble and 
disassemble it, with no tools or soldering required. A modular docking system enables 
you to pair it up with a variety of keyboards and gamepads, and you also can wear it 
on your wrist or in a holster clipped to a belt or pocket. Noodle Air is an air-gapped 
version of Noodle Pi, intended for high-security applications. 


Noodle Pi was launched on Kickstarter in July 2017, and it has shipped to backers in 
17 countries. Sadly, the HyperPixel display used in the Noodle Pi was discontinued less 
than a year after launch, so the original Noodle Pi is currently unavailable. An updated 
version is in the works and should be available soon. 


I’m planning to launch a Kickstarter for a different Noodle Pi device very soon (not 
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the updated version of the original Noodle Pi, which will be next). 


BL: When you first started working on the Noodle Pi, did you always envision that it’d 
be based on a Raspberry Pi Zero? 


AG: In July 2016, when | decided to make a handheld device as a deployment platform 
for Unsnoopable (my app for completely unsnoopable messaging), | envisioned that 
device as based on the Pi Zero right from the start. 


But I’ve been trying to put together a practical, open and flexible wearable computer 
for almost two decades. 


Around 2012, | hacked together a few prototype wearable Noodle computers for my 
own use. Some of them were based on the original Raspberry Pi Model B released in 
February 2012. 


My first wearable Linux computer, back in 2001, was based on a Sony VAIO 


Picturebook (with a Transmeta Crusoe processor!) , a Daeyang CyVisor head-mounted 
display and a Twiddler chorded keyboard. 


BL: Did you consider any other boards back then? 


AG: The Pi Zero was pretty much the only board small enough to fit into a reasonably 
small handheld, with the ability to run a full Linux system and for which some 
compatible touchscreen displays were readily available. | also wanted to include a 
camera in the device, and the Pi Zero v1.3, which included a camera connector, had 
just been released. So that was perfect. 


| did later consider the Pi Compute Module 3 Lite, which has a faster processor than 
the Pi Zero. However, that would need a carrier board, and together they’d end up 
being too big. 


The original Noodle Pi design in July 2016 also required a second board, or additional 
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components soldered onto the Pi Zero’s USB pads, to add WiFi and Bluetooth 
support, which made it very difficult to cram everything into the form factor of device 
| was aiming for. 


So for awhile, | procrastinated on the Noodle Pi and worked on some other 
prototypes based on the Jaguar Board. The procrastination strategy worked, and 
ideas from those prototypes helped with the design of the Noodle Pi. 


Procrastinating on the Noodle Pi also paid off when the need for additional 
components for wireless functionality was neatly eliminated in February 2017 with the 
launch of the Pi Zero W, which added onboard WiFi and Bluetooth. 


The touchscreen was the final sticking point as none of the small screens available 

in early 2017 offered a reasonably high resolution, and they required soldering and 
destructive modifications to fit into a reasonable device form factor. That was solved 
in June 2017 with the HyperPixel display. 


BL: Did the Pi Zero meet your needs? Was it...fast enough? 


AG: Yes, | think the Pi Zero is a fantastic little SBC, and it was very cool how it got 
the camera interface and the WiFi/Bluetooth upgrades just as | needed them for 
the Noodle Pi—and the HyperPixel display. With all these capabilities, | think it’s 
easily the most versatile and feature-packed SBC of its size. In 2018, the Pi Zero 
WH version was launched, which includes a pre-soldered header. That simplifies 
the assembly of the Noodle Pi. 


Speed definitely is a bit of a constraint on the Pi Zero, but then my main goal was to 
deploy fairly lightweight apps written in Perl, and the Pi Zero is more than capable 
of handling that, and of scanning QR codes using its camera, which is important for 
some of my apps. 


Compared to the first computer | owned, an IBM PC XT with an 8088 @ 4.77 MHz and 
640KB of RAM, the Pi Zero is a blazing speed demon. It’s great for console apps, and it 
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runs most GUI apps just fine too. 


Of course, it isn’t really suited for some applications, but the great thing is that you 
can just pull out your MicroSD card from a Pi Zero, pop it into a Pi 3, and you have 


the same system running on a much faster computer. | use this trick quite often. 


You can use a Noodle Pi on the go, and at home/office, just pop the MicroSD into a Pi 
3 connected to a big screen and full keyboard, and presto, it’s a faster more powerful 
desktop machine! 


BL: If you had it to do all over again, what sort of single-board computer would 
you use? 


AG: For asmall integrated handheld device, I'd stick with the Pi Zero. ’m working on 
the updated version of the Noodle Pi, which should be available in the near future. 


’m also working on a slightly different variation of the Noodle Pi, which | plan to 
launch first, probably in February 2019. 


| picked up some NanoPi SBCs to play around with. There are some interesting 
possibilities there, but they don’t have integrated WiFi/Bluetooth antennas like the Pi 
Zero W does, so the Zero still wins for wireless connectivity. 


There’s also the Banana Pi Zero, which could potentially be a nice drop-in upgrade for 
the Pi Zero, but | wasn’t able to get it to boot up anything the last time | tried. And it 
doesn’t have integrated wireless antennas either. 


BL: What part of building something around a single-board computer was the least 
enjoyable? Any headaches? 


AG: For the most part, it was and continues to be quite enjoyable. | love working with 
SBCs, and it’s great that there’s such a variety of them available now. One thing | don’t 


particularly enjoy is soldering, which is why | wanted to make sure that the Noodle Pi 
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could be assembled without any soldering required. 


Back in the early 1990s, when | got started with digital electronics, | enjoyed making 
prototype circuit boards using wire-wrapping. That was great fun. | made a whole 
custom 68K-powered computer that way. 


With the original Pi model B in 2012, the USB ports were quite flaky and that was a 
pain, but the Pi Zero’s USB is fine. 


The headaches were more on the production side of things, with endless tiny tweaks 
and prototyping required to get the model and print parameters just right, and with 
various malfunctions and maintenance issues with the 3D printers during production. 


’m hoping we'll see some significant advances in consumer 3D printing tech soon, 
leading to big reliability and quality improvements. 


BL: Do you use Pis (or other similar boards) in your own life, outside the Noodle Pi? 


AG: Absolutely. In fact, | now rely on SBCs for nearly all of my computing (other than 
production servers). | have quite a variety of prototype Noodle devices based on Pis 
and other SBCs, which serve as my main workstations and mobile computers. | hope 
to launch some more of these as Noodle products over the next few months. 


| also have a bunch of boards in clusters for testing the HashCash vault and some 
other projects. 


’'ve used every conceivable type of mobile computer during the past 25 years, and I’ve 
never really found the perfect sweet spot of mobility, price and power with laptops 
and commercially available handheld devices, which is what led me to keep hacking up 
Noodle computers. 


| think the modularity afforded by SBCs is a big win. With Noodle computers based on 
SBCs, | don’t have to worry about a screen or battery or keyboard problem rendering 


my whole machine unusable. | can just quickly swap out parts and get back to work. 


107 | March 2019 | https://www.linuxjournal.com 


DEEP 
DIVE 


ve had work disrupted way too many times because of such issues with laptops, 
which could be repaired only by the authorized repair center. That’s no good for me. | 
need computers that | can fix myself in five minutes. 


This is why | haven’t been tempted to pick up a GPD Pocket or the Gemini PDA, 
which | would have paid big bucks for five years ago. After switching mainly to Noodle 
computers based on SBCs, | find that the modularity and repair-ability are just too 
important to give up. 


BL: If you could change one thing about the Pi Zero, what would it be? 


AG: If a genie granted me one change to the Pi Zero, I’d say: make it completely open 
with no proprietary blob required to boot up. 


Of course, it’d also be great to see it updated to a faster multi-core CPU. But for 
the most part, | think it’s an awesome SBC that packs incredible capabilities into a 
really tiny package. 


BL: Any other anecdotes or interesting stories you'd like to share about the process 
of building a Pi Zero-based handheld? 


AG: This isn’t specifically about building a Pi Zero handheld, but | find it fun and 
interesting how procrastinating at various points allowed the available tech time to 
catch up to what | was trying to do. 


And given that my main goal with the Noodle Pi was just to be able to deploy my Perl 
apps on an open handheld device, had | procrastinated a bit longer, | probably could 
just have done that on the upcoming Purism Librem 5 and saved myself a lot of work! 


On the other hand, it’s nice to have had the Noodle Pi as a development platform 


during the past year and a half. Apps written for the Noodle Pi should be easy to 
deploy on the Librem 5 when it ships, hopefully with minimal changes to the code. 
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Conclusion 

Big round of thanks to both Pete Barker and Ashish Gulhati for taking the time 

to share their experiences in building devices based on the Pi Zero. | love seeing 
independent and creative makers designing (and shipping) cool little niche Linux 
computers like this. I’m absolutely positive I’m not alone in hoping that the coming 
years sees this trend increasing. ™ 


Bryan Lunduke is a former Software Tester, former Programmer, former VP of Technology, former Linux Marketing 
Guy (tm), former openSUSE Board Member... and current Deputy Editor of Linux Journal as well as host of the 
popular Lunduke Show. More details: http://lunduke.com. 


Send comments or feedback 
via http://www.linuxjournal.com/contact 
or email |jeditor@linuxjournal.com. 
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Mycroft: a Privacy- 
Respecting Digital 
Assistant 


How to build a Mycroft skill and then convert the Google AIY 
Voice Kit to run Mycroft instead of Google Assistant. 


By Jan Newmarch 


Mycroft is a digital assistant along the lines of Google Home, Amazon Alexa and the 
many others that are currently coming onto the market. Unlike most of these though, 
it has two major differences: 


- The code is open source, and you can install it on most Linux systems. 

- There is a focus on privacy. 

Many assistants will listen all of the time, sending everything they hear into the 
“cloud”. As you all know, many of these companies are vacuums for data, and their 
business models are built on using that data. Mycroft goes against that model by 
minimizing the amount of data collected and anonymizing the data when it has to 


interact with other systems. 


Google processing is given in Table 1. 
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Table 1. Google Assistant Query Processing 


Local Execution Remote Execution 


Speech to text (recorded in your Google 
account) 


ee 
ee 
es 
ee 
ee 


Mycroft changes this around to what’s shown in Table 2. 


Table 2. Mycroft Query Processing 


Local Execution Remote execution 
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ee 
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All processing is local except for the speech to text and text to speech modules, and 
possibly an external web service. The “speech to text” service is currently Google’s 
Speech to Text service, but there is an option of using the Mozilla DeepSpeech engine. 
The requests are anonymized to be from “MycroftAl” rather than from a particular 
user. The “Remote service if required by skill” will depend on what needs to be done to 
satisfy the query—for example, getting the local time won’t need one, while getting the 
weather will need access to a weather service, such as OpenWeatherMap. The “Text to 
speech” module, by default, uses a mechanical-sounding voice produced by software 
called Mimic 2, based on the Tacotron architecture, but the voice is configurable. 
However, there is a noticeable lag of several seconds while a voice response is 
prepared, and choosing a more complex voice increases this delay. 


The Google AIY Voice Kit 


You can buy digital assistants as “black boxes”. Google has created a do-it-yourself 
“white box” (Figure 1). It consists of the following: 


Figure 1. 
Google AlY 
Voice Kit 
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« Acardboard enclosure (yes okay, it’s brown, not white). 

- A speaker. 

* A microphone. 

- A “voice hat” board to process audio. 

- A big button to give some privacy—the kit won’t listen until you press the button. 
The kit is driven by a Raspberry Pi 3, which you supply yourself. 


You can download an image for the Raspberry Pi that can drive the voice hat, and 
there’s a lengthy set of instructions to turn it into a Google Assistant. This is non- 
trivial, requiring a Google account and using OAuth for identification purposes. But 
I’m not going that route here, as this is a nice little box for installing Mycroft. 


Installing Mycroft 

Mycroft requires a good quality microphone and reasonable speaker, which the 
Raspberry Pi doesn’t have. The AIlY kit has these as well as the little brown box 
in which to put everything. But, the AIY image doesn’t include Mycroft, and the 
Raspberry Pi image for Mycroft—picroft—doesn’t have drivers for the voice hat. 


There are two ways if getting Mycroft onto the Raspberry Pi you use with the AIY kit: 
add Mycroft to the AIY image or install AIY support onto the picroft image. 


The first is straightforward but time-consuming: using the AlY image from here, 
download the Mycroft files from GitHub and build it on the Raspberry Pi. It’s not hard. 


The second is slightly less time-consuming: the Raspberry Pi image from September 
2018 allows you to select the AIY devices during setup, and then downloads and 
builds all the relevant AIY files. At the time of this writing, Google has broken this by 
not having a particular Python “wheel” file available for the Raspberry Pi 3B or 3B+, 
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but there is a workaround from andlo here. Hopefully this will be fixed soon—the AIY 
forums are very voluble on the subject! 


To build this kit, follow the AIY instructions to connect everything together. But, don’t 
fold them all into the cardboard enclosure yet, as that hides the USB ports and the 


HDMI port. Connect an external monitor, keyboard and mouse, so you can control 
the system while you build it. From then on, follow these steps: 


- Using the AlY image: download and build Mycroft from GitHub. 


- Using the picroft image: select the AIY option during setup, which should 
download and build lots of Python wheel files and libraries. While this is broken at 
the time of this writing (December 2018), hopefully Google will have fixed it by 
the time you read this. 


Figure 2. Screenshot of Mycroft in debug mode—this shows the skills loaded at the top and 
the interactions with the user at the bottom. 
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Running Mycroft 

Once booted, you can run Mycroft from the directory mycroft-core using the 
start-mycroft.sh command. This can take parameters, and the most voluble way 
of seeing what’s going on is to run it in debug mode: 


start-mycroft.sh debug 


This brings up the curses screen shown in Figure 2. 


Building Your Own Skills 


Mycroft has an ever-growing list of skills that come by default. These include Pandora, 
Spotify, OpenHAB and Wikipedia. But there’s always room for more! I’ve got some 
smart bulbs in my house, and I’d like to control them using Mycroft. 


Adding a skill means you need to know how to talk to the service you’re requesting. 
Often this information isn’t publicly available and has to be reverse-engineered. Such 
is the case with the Lanbon light switches, for example, but that’s another story! 
One device that is okay is the Yeelight smart light. It has the advantage of having a 
documented “local” API to control it, so there’s no need to tell the Yeelight server 
what time you go to bed, for example. The Hue and LIFX lights offer similar privacy- 
respecting APIs; 


The Mycroft skills are in the /opt/mycroft/skills directory. There is one directory per 
skill, and the following sub-directories and files: 


- Avocab directory per language, such as vocab/en-us for US English phrases that 
will trigger an intent. 


- Adialog directory per language, such as dialog/de-de for German language 
responses. 


* The Python file init__.py with a class for the skill and methods for each of the 
intents. 
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« Some additional files for non-basic features. 


The __init__.py file is where all the work of the skill is done. It needs to import 
required classes and define a class that inherits from MycroftSkill. This class 
should have methods for each intent, which is triggered by the keywords for that 
intent. This was formerly signaled by code in the constructor, but the preference now 
is for an adornment on each method. The method then does what is needed, and 
finally calls for a response to be sent. 


The Yeelight Bulb Skill 


| use the Yeelight bulbs here, as they 
are relatively simple, documented, 

and it’s easy enough to show how to 
build a small skill—note that there are 
additional expectations of a real skill 
that would be distributed by Mycroft! 
Mycroft has a more detailed set of 
instructions at Developing a new Skill. 


Yeelight makes a number of smart 
bulbs that you can control through an 
Android or iPhone app. Like many home 
loT devices, it Communicates with a 
(Yeelight) server, and to set it up, you 
have to go through the usual complex 
registration processes for smart-home 
devices. But, it also has a REST-like API 
through which you can control it with 
appropriate network calls. Typically, 
these would be restricted to the home 


LAN unless ports are explicitly opened BATE 


on the home router (bad idea!). But 
if you just want to control the lights Figure 3. Enabling LAN API of the Yeelight 
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from inside the house, this is a good way. The “Yeelight WiFi Light Inter-Operation 
Specification” is described here. 


The default is for LAN control to be disabled for each bulb. You have to enable the 
“Control LAN” mode, which is found by selecting a Yeelight device, scrolling down to 
the bottom of the screen and choosing the rightmost icon and then the Control LAN 


icon. The rather crude screenshot from Yeelight in Figure 3 shows this. 


There are a number of different Yeelight bulbs. | look only at the common feature of 
on/off for each bulb in this article. The Mycroft files for such a skill are as follows: 


- Yeelight/vocab/en-us/OnKeywords.voc 
- Yeelight/vocab/en-us/Offkeywords.voc 


> Yeelight/dialog/en-us/on.activity.dialog 


> Yeelight/dialog/en-us/off.activity.dialog 
- Yeelight/__init_.py 
- Yeelight/requirements.txt 


After enabling LAN control, the Yeelight bulb listens to multicast messages 
to 239.255.255.250 on port 1982. On receiving the correct message, each 


bulb responds with a packet containing information such as “Location: 
yeelight://192.168.1.25:55443”. Several libraries have been developed to do this and 
other Yeelight commands. | use the library by Stavros Korokithakis available here. 
This can be installed by pip, but Mycroft will do it for you: the file requirements. txt 
contains a list of the packages needed to run a skill, which will be loaded by Mycroft, 
so here will contain the name of the Yeelight package: 


yeelight 
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The package contains a method discover_bulbs(), which sends out the multicast 
message and collects the responses as a list, each element containing the IP address 
and port, and a list of the light’s capabilities. A new Bulb() can then be created using 
the IP address. The simplest logic finds a list of bulbs and returns the first in the list, 
along with its name: 


def select_light(location): 

yeelights = yeelight.discover_bulbs() 

if len(yeelights) >= 1: 
light_info = yeelights [@] 
Lo Light: inte [ip] 
name = light_info['capabilities'] ['name'] 
return light, name 

return None, None 


Once a light is found, commands can be sent to it. A command is in the form of a 
JSON string, which is constructed representing the command and its parameters and 
sent as a TCP packet to the bulb. These are encapsulated by the Yeelight package by 
methods including turn_on() and turn_off(). 


Finally, you can give the “on” command, which combines the decorator to register the 
intent, the command itself and the response. The response contains the light’s name, 
if it is available, to show that parameters can be passed to the speech dialog: 


@intent_handler(IntentBuilder("OnIntent").\ 
require("OnKeywords").build() ) 
def on_activity_intent(self, message): 
“Turn on Yeelight 
light, name = select_light( location) 
if light != None: 
light.turn_on() 
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if name == None: 
report = {"location": "unknown"} 
else: 


report = {"location": name} 
self.speak_dialog("on.activity", report) 


The (US English) intent matcher for switching the light on is in the file Yeelight/vocab/ 
en-us/Onkeywords.voc: 


Light ‘on 
And the (US English) response is in the file Yeelight/dialog/en-us/on.activity.dialog: 
turning on {{location}} 


If the light’s name is, say, “bedroom”, Mycroft will say “turning on bedroom”. There 
is considerably more complexity that can be built in to the intent matcher and 
responses, these are just indicative. 


Other Home Devices 


Just considering lighting, many options exist now. Let’s look at a few of them. 


LIFX An API for both LAN and WAN management of devices is documented. The 
LAN interface uses UDP messages only, and the different devices are distinguished by 
their MAC address rather than IP address. The WAN API sends HTTP REST commands 
via the LIFX server. Several Python packages implement this, such as lifxlan for LAN 
control and pifx for WAN control. There is a Mycroft skill using the WAN API. 


Hue Hue lights do not live on the IP network, but on a Zigbee network. The 
interaction with the LAN is via a Hue bridge: you talk IP to the bridge, it talks Zigbee 


to the devices. A REST API allows you to send requests to a light by a request to 
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the bridge including the light’s ID. Many Python packages implement this, and there 
already is a Mycroft skill for these lights. 


IKEA Tradfr The IKEA lights also use Zigbee via a gateway. There is an API using 


CoAP over UDP, and it has been implemented in a Python package called pytradfri. 
Although there is no Mycroft skill for these lights, there is an OpenHAB binding and a 
Mycroft skill for OpenHAB. 


Eufy Lumos This is an |P bulb, but the API does not appear to have been published. 
There have been several attempts to reverse-engineer the API, and there is currently a 
minimal Python package lakeside. 


Note: although the functions performed by each smart light are similar, there is 
little consistency in their networking APIs, and there are a variety of Python and 
other language implementations. Unfortunately, many vendors have not published 
their APIs. This is reminiscent of the early days of device drivers for Linux, where 
some vendors were good, and others stupidly thought themselves clever for hiding 
specifications. A vendor of one product has told me they will give me the API if | 
purchase 2,000 units at about $100 each—not on my salary! 


Security 
There are multiple security issues to be considered in any loT system, and several of 
them surface here. 


First, there are security issues arising from switching the bulb to local access mode: 
it means that anyone on your local network (such as the disgruntled teenager you’ve 
just sent off early to bed) can program your lights to flash on and off just as you 

get to sleep. In the case of the Yeelight, that’s a choice you get to make, as you have 
to change its mode explicitly. ’m assuming, of course, that your wireless network 
has security, such as WPA2 turned on, and any default router passwords have been 
changed—otherwise, you are vulnerable to any external attack! 


You can reduce the security risks by running these lights on their own subnet, even 
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ona different SSID with an appropriate firewall. This increases network configuration 
complexity as well as requiring a router that will pass multicast searches from the 
Mycroft subnet to the light’s subnet. The IETF expects home networks to have 
multiple IP subnets in the future (at RFC 7368: IPv6 Home Networking Architecture 
Principles), so such issues will need to be addressed in user-friendly ways eventually. 


Of course, you could say that this is all irrelevant, since anyone can just ask Mycroft 
to turn the lights on or off. Physical presence now clashes with network security! The 
Mycroft wake word recognizer cannot yet distinguish between different voices, but 
that’s in the works. 


What's Next? 


Mycroft will continue to support the open-source version, and it’s building up 
the skill sets. The company also makes its own hardware version, and version 2 
will be released soon. This will have its own FPGA, which will allow more Al-style 
processing to be done onboard. So Linux hackers can play around to their hearts’ 


Resources 


- Abraham Kang, “Understanding the Differences Between Alexa, API.ai, 
WIT.ai, and LUIS/Cortana” 


- Fox News “Google is recording everything you say to a bot—right now” 


« Steve Penrod, “Why We’re moving to DeepSpeech on March 31 | Privacy, 
Speech to Text & Balance” 


- Mycroft.ai, Mimic 
- Mycroft.ai, Mycroft Al Open Source Voice Assistant 


- Google AIY Voice Kit 
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content, and privacy-concerned individuals and companies can purchase an 
off-the-shelf version. 


The main downside is the lack of specifications and stability for services and loT 
devices. This is shown by Logitech in December 2018 turning off an undocumented 
LAN mode for the Harmony Hub due to security issues with its XMPP code. It caused 
an outcry from those relying on it for their home automation systems, and Logitech is 
now reconsidering its position—at least the company is listening; others are not! m 


Disclaimer: the author recently purchased shares in Mycroft. 


Jan Newmarch has written several articles for the Linux Journal in the past, as well as more than 80 papers and 
six books. He is Professor of loT at Box Hill Institute, and also Adjunct Professor at the University of Canberra and 
Adjunct Lecturer at Charles Sturt University. He expects to retire soon to a life of leisure, hacking and, of course, 
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Oracle Linux 
on Btrfs for the 
Raspberry Pi 


Enterprise comes to the micro server. 


By Charles Fisher 


Oracle Linux 7 has been released for the Raspberry Pi 3. The release packages 
Btrfs as the root filesystem on the UEK-branded Linux 4.14 Long Term Support 
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(LTS) kernel. A bootable disk image with a minimal install is provided along with a 
standard ISO installer. 


CentOS appears to support only the “Mustang” Applied Micro X-Gene for AArché4, 
and it provides the older AArch32 environment for all models of the Raspberry Pi. 
Oracle Linux is a compelling option among RPM distributions in supporting AArch64 
for the Pi Model 3. 


This is not to say that Oracle AArch64 Linux is without flaw, as Oracle warns that 
this is “a preview release and for development purposes only; Oracle suggests these 
not be used in production.” The non-functional WiFi device is missing firmware and 
documentation, which Oracle admits was overlooked. No X11 graphics are included 
in the image, although you can install them. The eponymous database client (and 
server) are absent. Oracle has provided a previous example of orphaned software 
with its Linux for SPARC project, which was abandoned after two minor releases. 
There’s no guarantee that this ARM version will not suffer the same fate, although 
Oracle has responded that “our eventual target is server class platforms”. One 
possible hardware target is the Fujitsu A64FX, a new server processor that bundles 48 
addressable AArch64 cores and 32GB of RAM on one die, asserted to be the “fastest 
server processor” that exists. 


AArch64 on the Pi 


You'll need a Raspberry Pi Model 3 to run Oracle Linux. The 3B+ is the best available 
device, and you should choose that over the predecessor Model 3B and all other 
previous models. Both Model 3 boards retain the (constraining) 1GB of RAM—a 


SODIMM socket would be far more practical. The newer board has a CPU that is 
200MHz faster and a Gigabit-compatible Ethernet port (that is limited to 300Mbit due 
to the USB2 linkage that connects it). A Model A also exists, but it lacks many of the 
ports on the 3B. More important, the Model 3 platform introduces a 64-bit CPU. 


ARM was very tardy to 64-bit addressing capabilities compared to other well known 
microprocessor families, announcing this extension in 2011. Intel’s first attempt to 
migrate the x86 market to the abortive Itanium 64-bit architecture shipped in 2001, 
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ultimately yielding to AMD64, which debuted in 2003. MIPS and SPARC made this 
transition much, much earlier (1991 and 1995, respectively). 


Despite the late arrival, the ARM AArch64 CPU architecture is now the dominant 
mobile platform. All supported iPhones are now required to run it, and most modern 
Android devices have migrated to it. ARM has guarded a competitive edge in power 
efficiency, as vast changes in its instruction set ideology and implementation have 
allowed ARM to maintain its market leadership in mobile. 


The Acorn/Advanced RISC Machine (ARM) began with the retroactively named 
AArch32 assembly and machine language that was designed by Furber and Wilson 
for the Archimedes desktop computer, where tremendous power efficiency “was 
a complete accident”. Desktop performance remained an architectural focus for a 
decade after the birth of ARM. 


Apple’s decision to base the (failed) Newton on ARM opened a new market of 
mobile device applications that prompted a new instruction set for ultra-compact 
assembly—Thumb 1 and 2. These are distinct from AArch32 and AArché4, and they 
focus on code density and minimal footprint for mobile devices. Thumb 2 is a 16-/32- 
bit variable length instruction set, which is dynamically translated to AArch32. Many 
other ARM extensions exist, but Thumb appears to have both great flexibility and 
persistence across ARM implementations. 


When mobile devices neared the 4Gb RAM limit of 32-bit ARM, the designers decided 
to break from the past. The primary mistakes in AArch32 have long been known: 


Design errors, like having r15 as the program counter or making every 
instruction conditional, are problems for CPU architects rather than 
programmers, and it’s no surprise that they disappeared in the 64-bit version 
of the ARM architecture. They must have made it awfully hard to implement 
superscalar, out-of-order execution. 


The changes in AArch64 bring it much closer to the spirit of MIPS, the most notable 
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of which are: 


- Conditional Execution has been removed, facilitating out-of-order processing 
in multiple pipelines. 


- The R15/PC register can now be manipulated only by a small number of 
jumping and branching instructions, vastly simplifying branch prediction. 


These performance improvements, along with the increase of pointer sizes to 64-bits, 
come at the cost of code density—programs compiled for native AArché4 will be 


larger than the equivalent for AArch32. Despite these enhancements, the majority of 
Intel desktop processors of the last decade easily will beat the Pi in most benchmarks 
(but they will not do so with a 10-Watt power supply). | examine the code density 
impact of AArché4 in greater detail below. 


Installation 

Most Raspberry Pi users rely on flash memory, which comes in two grades. Multi- 
Level Cell (MLC) media is cheap and offers large amounts of storage, but it can decay 
very quickly (cells typically are destroyed after 5,000 write operations). Most retail 
flash media (SD cards, USB flash drives) are MLC, and will not have a long lifetime 
under high I/O usage, despite the “wear-leveling” electronics that attempt to distribute 
writes over the whole device evenly. Single-Level Cell (SLC) media is more expensive 
and offers smaller amounts of storage, but it drastically increases the number of write 
operations until cell failure (100,000). Both types of memory can be “rehabilitated” 
by heating them, but this is not feasible for memory in most plastic packaging. If you 
anticipate large amounts of I/O, plan to purchase the correct grade of flash. 


One great benefit of the new Model 3 is the ability to boot from USB. A standard 
hard disk drive is now a boot option. SLC media is also more plentiful and inexpensive 


in the USB flash format than as microSD cards. Choose the format that fits your 
expected I/O usage. 


Once you have selected and obtained your media, you’re ready to download and 
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uncompress the following file: 


$ xz -dkv rpi3-ol7.6-—image-20181116. img. xz 
rpi3-ol7.6-image-20181116.img.xz (1/1) 
100 % 266.4 MiB / 5120.0 MiB = @.052 55 MiB/s 1332 


The full size of the boot image is 5GB—your boot media must be at least this size: 


$ LL rpi3—-ol7.6-—image—20181116. img* 

=rw-r-—-r—— 1 root root 5368709120 Jan 13 18:52 
rpi3—-ol7.6-—image—20181116. img 

=fw=r==r=— 1 root root 279309592 Jan 15 18252 
rpi3-ol7.6-—image—20181116. img. xz 


Insert your boot media and ensure that it is detected, but not mounted: 

# dmesg | tail -3 

[ 378.540649] mmc®: new high speed SDHC card at address 0002 
[ 

[ 


378.544104] mmcblkO: mmc0:0002 00000 7.83 GiB 
378.548395] mmcblkQ: pl 


Finally, write the image to the raw media: 
# dd if=rpi3-ol7.6-—image-20181116.img of=/dev/mmcblk®@ bs=4M 


Assuming there are no write errors, the media is now prepared to boot the Raspberry 
Pi Model 3. 


Operation 

Load the media into the Pi, connect an HDMI cable to a monitor, and attach an 
ethernet cable. After the power supply is connected, the Pi will boot (there is no 
power switch). 


127 | March 2019 | https://www.linuxjournal.com 


DEEP 
DIVE 


The Pi might fail to boot with older USB peripherals. When a well-worn IBM 89P8800 
USB 1.1 keyboard was attached, the firmware issued the message Timeout poll on 
interrupt endpoint and refused to boot. Trying a slightly newer Lenovo 41A5248 
keyboard, the boot proceeded but was greatly delayed. Both keyboards worked 
without error once the OS was running. It might be wise to boot initially without any 
non-essential USB hardware. 


Once the boot is complete, a login: prompt should be displayed. The user root 
with the password oracle will allow you to select a new root password, then drop to 
Bash. 


The /proc/cpuinfo file will list four processor cores with the following descriptions: 


[root@rpi3 ~]# cat /proc/cpuinfo 


processor Poti Eee 2 mie. 

BogoMIPS : 38.40 

Features : fp asimd evtstrm crc32 cpuid 
CPU implementer : @x41 


CPU architecture: 8 
CPU variant : 0x@ 

CPU part : QxdQ3 
CPU revision : 4 


The root filesystem is on Btrfs, which is a change from the XFS that you normally 
see on the x86_64 (AMD64) version of Oracle/Red Hat/CentOS/Scientific Linux 7. 
The /boot filesystem is on EXT4, likely due to bootloader considerations: 


[root@rpi3 ~]# mount | egrep ‘btrfs|ext4' 

/dev/mmcblk@p4 on / type btrfs 
(rw,noatime,ssd,space_cache, subvolid=5, subvol=/) 
/dev/mmcblk@p2 on /boot type ext4 (rw,noatime, data=ordered) 


Observe also the ssd mount option above. Btrfs detected this option automatically 
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and was previously unsafe with a “a negative impact on usability and lifetime” 

of the flash media, but it’s now appropriate in the 4.14 kernel. Observe that the 
autodetected SSD mount option was not specified in /etc/fstab (I’ve removed the 
UUIDs and LABELs): 


[root@rpi3 ~]# sed -r 's/*(UUID|LABEL) [* ]*x[ ]*//' /etc/fstab 
#Generated by RootFS Build Factory 

/boot/efi vfat noatime @ @ 

/boot ext4 noatime Q 0 

Swap Swap noatime 0 0 

/ btrfs noatime Q 0 

tmpfs /tmp tmpfs rw, nodev,nosuid,Size=128M @ @ 


The root filesystem is on the p4 partition of the SD card: 
[root@rpi3 ~]# fdisk -1l 


Disk /dev/mmcblk@: 7948 MB, 7948206080 bytes, 1552384@ sectors 
Units = sectors of 1 * 512 = 512 bytes 

Sector size (logical/physical): 512 bytes / 512 bytes 

I/O size (minimum/optimal): 512 bytes / 512 bytes 

Disk label type: dos 

Disk identifier: 0x000164f6 


Device Boot Start End Blocks Id System 
/dev/mmcblk0p1 2048 526335 262144 c W95 FAT32 
+ (LBA) 
/dev/mmcblk@p2 526336 1550335 512000 83 Linux 
/dev/mmcblkQ@p3 1550336 2074623 262144 #82 Linux 
oswap / Solaris 
/dev/mmcblk0p4 2074624 10463231 4194304 83 Linux 


Note above that I’m running on an 8GB SD card, but the last third of the card is 
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unused, as it does not lie in a partition. You can add the unused space to the root 
filesystem by first expanding the partition: 


[root@rpi3 ~]# growpart /dev/mmcblk®@ 4 
CHANGED: partition=4 start=2074624 old: size=8388608 
¢end=10463232 new: size=13449183, end=15523807 


And then expanding the Btrfs filesystem into the newly allocated space: 


[root@rpi3 ~]# btrfs filesystem resize max / 


Resize */* of ‘max 
The root filesystem now occupies the rest of the flash device: 
[root@rpi3 ~]# fdisk -l 


Disk /dev/mmcblk@: 7948 MB, 7948206080 bytes, 15523840 sectors 
Units = sectors of 1 * 512 = 512 bytes 

Sector size (logical/physical): 512 bytes / 512 bytes 

I/0 size (minimum/optimal): 512 bytes / 512 bytes 

Disk label type: dos 

Disk identifier: 0x000164f6 


Device Boot Start End Blocks Id System 
/dev/mmcblk@p1 2048 526335 262144 c W95 FAT32 
+ (LBA) 
/dev/mmcb lk@p2 526336 1550335 512000 83 Linux 
/dev/mmcb lk@p3 1550336 2074623 262144 4x82 Linux 
oswap / Solaris 
/dev/mmcblk0p4 2074624 15523806 6724591+ 83 Linux 


Btrfs is an extremely powerful filesystem, similar in capabilities to ZFS. It’s capable 
of transparent compression, mirroring, error detection and it has self-healing 
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capabilities. (Watch for a future article on in-depth Btrfs coverage.) Oracle Linux 


on the Raspberry Pi provides a useful learning environment for many new tools and 


features, and the addition of Btrfs is chief among them. 


A number of missing utilities are present in minimal installs on x86_64. In no particular 


order of preference, some are ethtool, less, man and nmtui. Assuming internet 


connectivity to Oracle,a yum install man will bring in less as a dependency (and 
let you begin reading all the Btrfs manual pages). The yum whatprovides command 


is useful for searching the contents of uninstalled packages for a particular utility 


Busybox is an alternative to some of the native Oracle AArch64 packages. Those unfamiliar 
with Busybox might review my previous container article published in Linux Journal that 
details its use. The 1.28.1 release offers several ARM binaries of Busybox (listed below): 


[root@rpi3 ~]# for x in busybox-armx 


do ls -l $x; 


-rwxr-xr-x 1 root root 1132724 
busybox-armv5L: ELF 32-bit LSB 
e(SYSV) ans 


BusyBox v1.28.1 (2018-02-15 14: 


—-rwxr-xr-x 1 root root 836560 
busybox-armv7m: ELF 32-bit LSB 
o(SYSV)... 


BusyBox v1.28.1 (2018-02-15 14: 


—-rwxr-xr-x 1 root root 1079156 
busybox-armv7r: ELF 32-bit LSB 
o(SYSV)... 


BusyBox v1.28.1 (2018-02-15 14: 


—-rwxr-xr-x 1 root root 1078504 
busybox-armv8Ll: ELF 32-bit LSB 
o(SYSV)... 


BusyBox v1.28.1 (2018-02-15 14: 
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file $x; 


./$x | head -1; done 


Jan 10 17:23 busybox-armv5l 
executable, ARM, version 1 


34:02 CET) multi-call binary. 
Jan 10 17:23 busybox-armv7m 
shared object, ARM, version 1 


34:02 CET) multi-call binary. 
Jan 10 17:23 busybox-armv7r 
executable, ARM, version 1 
34:02 CET) multi-call binary. 
Jan 10 17:23 busybox-armv8l 
executable, ARM, version 1 


34:02 CET) multi-call binary. 
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Notice above that the busybox—armv7m binary is substantially smaller than all 
the rest. This binary is apparently composed of Thumb 2 code; the ARM 7 M and 
R architectures appear to exclude both AArch32 and AArch64: “ARMv7-M... No 
ARM instruction set support (Thumb only).” Thumb might explain the smaller 
size, but its use may come at some cost to performance. 


Oracle includes an AArch64 compiler environment, but this isn’t likely to be 
capable of emitting ARMv7-M code. Oracle doesn’t provide a glibc.aarch32 
or glibc.thumb2 package in the same way that it provides a glibc. i686 on 
AMD64, nor are there any 32-bit libraries in /usr/lib. ARM itself provides Thumb- 
capable GNU compilers, as do other sources. Using Thumb as a compiler target 


will conserve memory at the potential cost of performance. This might be a 
reasonable choice for standing deemons that are not CPU-intensive. 


One glaring lack in Oracle Linux on the Raspberry Pi is the missing WiFi device. The 
kernel dmesg has a clue to the problem: 


brcemfmac: brcmf_fw_map_chip_to_name: using 
brcem/brcmfmac43455-sdio.bin for chip 0x004345(17221) 
orev 0x000006 
usbcore: registered new interface driver brcmfmac 
brcemfmac mmc1:0001:1: Direct firmware load for 
brcm/brcmfmac43455-sdio.bin failed 
owith error -2 


You can find one source of the missing firmware at this link, although you also 
can find it within Raspbian. Installing the firmware will cause a wlanO device to 

appear, but all my attempts to configure it have failed. It doesn’t appear to be 

functional in the current release, despite the brcmfmac kernel module: 


[root@rpi3 ~]# cd /usr/lib/firmware/brcm/ 
[root@rpi3 ~]# 1l brcemfmac43455x 
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—-rw-r--r-- 1 root root 600487 Jan 14 19:33 
obrcmfmac43455-sdio.bin 

-rw-r--r-- 1 root root 14036 Jan 14 19:49 
obrcmfmac43455-sdio.clm_blob 

—rw-r--r-- 1 root root 2054 Jan 14 19:41 
obrcmfmac43455-sdio.txt 


lt appears that the WiFi and Bluetooth devices on the Raspberry Pi work 
through the SD card’s SDIO interface. Once those files are in place, reboot, 
and the WiFi driver should appear in the dmesg. Note this will use a small 
amount of additional memory: 


mmc1: new high speed SDIO card at address 0001 
brcemfmac: brcmf_fw_map_chip_to_name: using 
brcem/brcmfmac43455-sdio.bin for chip 0x004345(17221) 
orev @x000006 
brcemfmac: brcmf_c_preinit_dcmds: Firmware version = wlQ: 
Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) 
oFWID 01-4fbe0b04 
Bluetooth: Generic Bluetooth SDIO driver ver 9.1 


There is no mention at all of the WiFi hardware on the Raspberry Pi within 
Oracle’s documentation on the AArch64 release, which Oracle claims was 
an oversight. This also appears to be an issue in CentOS, where it is at least 
discussed at some length. 


Code Density 

As 1GB of RAM included on the Pi is constraining, you should have some idea of 
the penalty AArch64 imposes. 

Below is a script that I’ve used to size all the ELF binaries in Raspbian Linux 


running on the original Raspberry Pi, storing this in the file a32.txt: 
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for x in /bin/x 
do [ -f "$x" ] & 
case "$(file "$x")" in 
*ELFx) stat -c %n\ %S "$x"3; 
esac 
done > a32.txt 


Moving this file to Oracle Linux running on the Raspberry Pi Model 3 B+, | run the 
following to find the size differences: 


while read p s 
do [ -f "$p" ] & 

case "$(file "$p")" in 

*ELFx) echo $p $s $(stat -c %s "$p");; 

esac 

done < a32.txt | awk ' 
{at+=$2; b+=$3; print $1,$2,$3,$3/$2} 

END {print a,b,b/a}' > a64.txt 


For this small sample of 66 files, | found the results shown in Table 1. 


Table 1. Results of Size Differences of 66 Files 


Program Raspbian % increase 
/bin/bash O27 2 971728 1.06466 
/bin/cat 30560 70408 BOSS 
/bin/chgrp 51084 70944 1.36377 
/bin/chmod 46956 70840 1503865 
/bin/chown S107 71000 1.36965 
/bin/cp (C4522 204296 es e27 
[bin/cpio 118460 141752 1.19662 
/bin/date 83868 70368 0.839033 
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/bin/dd 63424 136456 2.15149 
/bin/df 67876 137848 2.03088 
/bin/dir 108804 138240 1.27054 
/bin/dmesg 59484 78296 ee 2s 
/bin/echo 26404 69904 2.64748 
/bin/false 22304 69880 3.13307 
/bin/findmnt a7 144 71992 1.38064 
/bin/grep 173656 204048 1.17501 
/bin/gzip 80476 137400 270734 
/bin/hostname 13964 69048 MO AAT | 
/bin/fjournalctl 63204 538448 Seed | 
/bin/kill 22020 70432 3.19855 
/bin/kmod 128560 203960 1.5865 
/bin/less 151382 219472 1.44969 
/bin/lessecho 9688 68752 7.09661 
/bin/lesskey 14460 70320 4.86307 
/bin/In 46976 70848 1.50817 
/bin/login SoA 70032 729055 
/bin/loginctl A 7 538280 12,5966 
/bin/ls 108804 138240 1.27054 
/bin/Isblk 67756 138336 2.04168 
/bin/mkdir 63472 137080 2.15969 
/bin/mknod 55248 71272 1.29004 
/bin/mktemp 34668 70288 2102745 
/bin/more 34708 69824 AON Ks: 
/bin/mount S437 2 68840 197408 
/bin/mountpoint 9896 68944 6.96686 
/bin/mv 100504 138480 1.37786 
/bin/netstat 106676 211912 oes 
/bin/ping 55720 70208 1.26001 
/bin/ps 83624 137000 n63e29 
/bin/pwd 26452 70056 2.64842 
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/bin/readlink 34628 70448 202447 
/bin/rm 51076 71056 1.39118 
/bin/rmdir 34628 70072 2.02356 
/bin/sed 84100 71904 0.854982 
/bin/sleep 26416 69984 DIAS S 
/bin/stty 59240 70240 1.18569 
/bin/su SC TS 69008 Ze AZ 
/bin/sync 26424 69912 264573 
/bin/systemctl 161680 738032 AE S647 7, 
/bin/systemd-ask- Novae 70000 7.05650 
password 
/bin/systemd- D236 69816 102657 
escape 
/bin/systemd-hwdb 67520 136520 2.02192 
/bin/systemd-inhibit 14040 33/7/28 24.0547 
/bin/systemd- 18128 69912 2S 5036 
machine-id-setup 
/bin/systemd-notify 9936 69728 TOT TA 
/bin/systemd- 50988 202912 5.2796 
tmpfiles 
/bin/systemd-tty- 26324 135920 SHiloses 
ask-password-agent 
/bin/tailf 22288 69488 Si77e 
/bin/tar 327644 350288 1069711 
/bin/touch 71584 70640 0.986813 
/bin/true 22304 69880 5.13307 
/bin/udevadm BES AS 469248 1, e696 
/bin/umount 22436 68856 SOs 
/bin/uname 26416 69928 2.64718 
/bin/vdir 108804 138240 1.27054 
/bin/wdctl 26408 70256 2.66041 
a1O076a2 9795488 oy Sl 
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These programs take nearly twice the space in Oracle Linux as they do in Raspbian. 
This somewhat explains the CentOS decision to remain on AArch32 with smaller 
32-bit binaries. Oracle’s pursuit of AArch64 is likely due to similar platforms that it 
supports or may support in the future. 


Should Oracle elect to provide a Thumb2 development environment in the same 
way that it supports 32-bit x86, then Oracle could produce even smaller binaries 
than are found in Raspbian while still running a 64-bit kernel, at some cost to 
performance. This assumes that all target platforms support Thumb2; by all 
appearances, the Fujitsu A64FX does not. 


It might be useful to examine commonly run server daemons, system libraries and 
extract the text/data/bss segment sizes within all of these programs to see greater 
detail on the AArch64 penalty paid here. Those with large ARM deployments are 
encouraged to do so. 


Conclusion 
It’s refreshing to have a new Linux distribution where legacy support is slashed 
in a way that never would be tolerated within Intel/AMD64 environments. There 


is substantial complexity and inertia in the maintenance of the systems of 
decades past. 


Still, the relative silence in the documentation on questions of (the overlooked 
WiFi) hardware support and the legacy Thumb and AArch32 instruction sets is 
unsettling. Operating system vendors should be clear on what their products 
can and cannot do with the target hardware. While there are issues with 

Oracle AArch64 Linux where that clarity is lacking, it must be conceded that 

this is a pre-production release, and the desired clarity and supported server- 
grade AArché4 target platforms may not yet exist. To reiterate, one possible 
hardware target is the Fujitsu A64FX, which designers are asserting as the fastest 
processor in the world. Amazon also recently began running ARM workloads 

in its EC2 cloud with its Graviton processor, but Gravitons are not expected to 
outperform the Fujitsu A64FX, and the relationship between Amazon and Oracle 
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is not warm. Oracle also may be developing its own AArch64 specifically tuned 
for Oracle’s database. Oracle previously has maintained SPARC in this capacity, 
and continues to dominate the TPC benchmark with it; the company also may 
decide to do this with its own ARM processor. 


On the subject of the Oracle Database, the absence of any discussion or mention 
of it also is a cause for substantial concern on the longevity of the platform. 


In any case, Oracle AArch64 Linux likely will be used by many pursuing low- 
power, large memory applications. The Raspberry Pi may be able to provide a 
development environment for those larger systems. It’s encouraging to see ARM 
move into the enterprise space, and the prospect of a legacy-free computing 
environment without all the problems (Meltdown), scandals (ME/PSP) and 
firmware concerns of Intel is quite refreshing. m 


Charles Fisher has an electrical engineering degree from the University of lowa and works as a systems and database administrator for 
a Fortune 500 mining and manufacturing corporation. 


Resources 


Oracle Linux for ARM Downloads 

Raspberry Pi (Wikipedia) 

Btrfs—the Next Generation Filesystem for Linux 
CentOS and AArch64 

Oracle Linux for Sparc Downloads 


“Fujitsu’s A64FX ARM Chip Waves The HPC Banner High” by Timothy 
Prickett Morgan 
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R4000 Microprocessor from 1991 (Wikipedia) 
UltraSPARC Microprocessor from 1995 (Wikipedia) 


“ARM Creators Sophie Wilson and Steve Furber, Part Two: 
the accidental chip” by Chris Bidmead, The Register 


Acorn Archimedes (Wikipedia) 

Apple Newton (Wikipedia) 

Thumb and Thumb-2 (Wikipedia) 

“RISC instruction sets | have known and disliked” by Jack Whitham 
Conditional Execution (Wikipedia) 

R15/PC register (Wikipedia) 

Types of Flash Memory (Wikipedia) 


“Making flash memory more reliable with 800°C heat pulses” 
by Mark Tyson 


Installation of Raspberry Pi 3 Image—Release Notes for Oracle Linux 7 
Update 6 (aarch64) 


Btrfs Gotchas 

“ZFS for Linux” by Charles Fisher, LJ, March 2018 

BusyBox 

“Infinite BusyBox with systemd” by Charles Fisher, LJ, March 2015 
BusyBox 1.28.1 Release 


What are the differences between ARMv7-A, ARMv7-R and ARMv7-M? 
(arm Developer) 


GNU ARM Embedded Toolchain Downloads (arm Developer) 


armbian firmware 
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PiBox: an Embedded 
systems Journey 


Raspberry Pi development with off-the-shelf software is easy by 
design. But, how would you use it to build a custom distribution 
with cross-compiled applications targeted for distributed media 
playback? Michael J. Hammel shares his experience in doing just 
that with his PiBox project. 


By Michael J. Hammel 


About 12 years ago, | started work on a system that was significantly different 
from the typical desktop/server systems I’d worked on previously. It was an 
embedded quad-core MIPS-based platform developed for general-purpose 
in-flight computational processing of radar data. That platform never saw 

the light of day, but it started me down a path that I’m still on today: building 
systems from scratch. And when | say systems, | mean the entire software stack 
from compiler toolchains through distributed end-user applications and into 
enclosures and production hardware. 


That early work led to a project | started in early 2011 called BeagleBox. 
BeagleBox is a custom build system aimed at the BeagleBoard low-cost 
Tl-based SoC board. The build system focus is on generating a cross-compiler 
toolchain that is used to compile bootloaders, the Linux kernel and an initial 
root filesystem. For a variety of non-technical reasons, the project switched in 
November 2012 to the Raspberry Pi, and the project is now known as PiBox. 


PiBox is a generic name that currently encompasses three platforms. The PiBox 
Development Platform is the core distribution. It aims to be an easily extendable 
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Ld xterm 


BusyBox v1,26,2 (2018-01-18 16:48:05 MST) built-in shell (ash) 
Enter ‘help’ for a list of built-in commands, 


*O 


BusyBox v1.26.2 (2018-01-18 16:48:05 MST) built-in shell (ash) 
Enter 'help' for a list of built-in commands. 


#fl 


Figure 1. The default UI for the PiBox developement platform is based on Blackbox. 


platform for purpose-driven applications using opkg package management. 

The development platform boots into a standard X desktop using Blackbox. 
Beyond the UI and network configuration, there is little functionality. The 
development platform is just a starting point, but it easily can be extended using 
the PiBox build environment. 


A collection of packages convert the Development Platform into a prototype 
consumer device called the PiBox Media Server. This device provides a simple 

Ul for navigation to services such as video playback. However, the device also 
provides services to distributed PiBox Media Players. The Players are software- 
identical to the Server, but they are used specifically to play remote video 
sources. Long term, you'll see the Media Server provide a cental hub for home 
automation via the lronMan project and form the platform for a DIY phone based 
on the Pi Zero. Components of the the Media Server have been used to create a 
kiosk picture and video player running on a Pi Touchscreen. 
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There are three primary goals for PiBox: 


1. Bea tool for learning not just the embedded development process but full systems 
development. 


2. Be easily extensible for purpose-driven applications. 
3. Be a prototype consumer device. 


Ancillary goals now include learning how to design and produce consumer-grade 
enclosures, power management and other hardware-related issues, parts supply and 
costing and production issues. 


PiBox met these goals with the current release, and the project continues to 
evolve by addressing what consumer devices really encompass. This includes 
support for intergration of mobile devices, voice command and control and 
distributed sensor networks. If you think of Linux as Linus’ answer to freely 
available UNIX and Minix, then PiBox would be the answer to building distributed 
loT infrastructure from the ground up. 


Media Server and Media Player 


The Media Server and Media Player are extensions of the Development Platform. 


The goal of the Media systems is to provide distributed media playback ina 
closed network, specifically in my family’s travel trailer. We replaced the 
TV/DVD with the Media Server, plugging in USB sticks filled with videos. The 
server distributes the videos to mobile devices, such as our Android and iOS 


tablets, but also to a Media Player. The player holds a Raspberry Pi connected 
to a pico projector, all enclosed in a box with audio port exposed. We plug in 
an audio splitter to headphones, point the player at the side of our trailer and 
watch movies at night while camping. 


The server, at the time of this writing, has limited support for Bluetooth devices, 
but it will eventually support an array of Bluetooth and BLE (Bluetouth low energy) 
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Figure 2. The Media Server and Media Player have specific hardware specifications. 


devices for monitoring sensors. There are many other long term goals for the server, 
including voice control, remote management and better support for a wider array of 
video formats. 


Hardware 

The Media Player is a self-contained box intended to run on battery power only. 
A Raspberry Pi is connected via HDMI to a handheld projector, a power switch 
and a battery. The Media Server contains a Pi connected to a 7-port UUGear USB 


hub, each powered through a single power switch connected to an external power 
supply. The Server exposes both the HDMI and audio ports, but the HDMI defaults 
to audio over HDMI. The HDMI doesn’t have to be connected for the server to 
function. The server USB ports are extended from the hub to the case to make 
them easy to access. 
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Figure 3. Media Server hardware includes enclosures with exposed USB ports. 


The design for both Server and Player specifies a single power-on switch. The current Player 
prototype requires powering the Pi and the projector separately, since the projector is an 
off-the-shelf model that would require modification to make it integrate cleanly. 
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The Build System 


At the root of PiBox is the core build system. There are many build systems for 
embedded systems. Many are based on bitbake, a Python-based metabuild system. 
Others embrace traditional GNU Make. Metabuilds are simply build systems that wrap 
other software builds. A metabuild system might include a toolchain, the Linux kernel 
and various pieces of a root filesystem. 


PiBox is a metabuild system that wraps four primary components: a cross toolchain 
for a chosen hardware platform, a bootloader, the Linux kernel and a root filesystem. 
The design goal of the metabuild system is to permit easy modification of the primary 
components, extend the primary components or change them to use different tools 
and provide a consistent set of targets for building them. 


The cross toolchain is built based on Crosstool-NG. For the Raspberry Pi, this includes 
the use of a custom glibc from Linaro, the ARM division devoted to improving software 
support for ARM devices. It also includes a custom GCC built with hard float support, 
allowing applications to take full advantage of the Broadcom processors they support. 


PiBox wraps Buildroot and BusyBox to generate a root filesystem. A small number 
of custom packages are added, including a standalone network configuration tool 
(PiBox Network Config and its associated libpne library). The root filesystem is 
designed to work with both wired Ethernet and WiFi out of the box. The core 
build system allows for easy rebuilds when changing the Buildroot configuration or 
testing Buildroot packages. 


The design of the build system is flexible enough to support the use of a bootloader, 
such as Das U-Boot. However, the Raspberry Pi is used with a binary bootloader, also 
referred to in this case as firmware, provided through a GitHub repository. PiBox 
downloads and unpacks this firmware during the bootloader phase of the build. 
However, it easily could be modified to use a bootloader that required compilation. 


The top-level Makefile includes files suffixed with .cfg and .mk from the config directory. 
The .cfg files provide configuration options that allow changing upstream repositories 
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and build options without having to modify build targets in the .mk files. The goal of the 
.mk files is to provide a consistent interface, so that all components are built using the 
same set of targets: -get, -unpack, -patch, -config, -pkg and so forth. Most important, 
all core components of the build offer -menuconf ig targets that allow changing the 
underlying tool configuration. All builds are done out of tree to keep build artifacts 
seperate from the build system. A -saveconfig target allows saving the changes to 
any component into the PiBox source tree. This makes it easier to commit to a local 

or upstream repository. A built-in help target explains all available component targets. 
Editing and saving a modified Buildroot configuration looks like this: 


make buildroot-—menuconf ig 
make buildroot-saveconfig 
make buildroot—clobber 
make buildroot 


PiBox does more than just build each component. The cross toolchain is relocatable. 
It can be packaged as an RPM that installs under /opt/rpiTC. The target components 
(bootloader, kernel and root filesystem) also are packaged along with scripts for 
formatting and populating an SD card. The SD card is used to boot the Raspberry Pi. 


The key advantage to the packaging is that the combination of the toolchain and a 
staging archive allow cross compiling of applications without having to build the core 
development platform. A project with multiple developers deploys the cross build 


environment to developer systems. Developers focus on app development instead 
of building the core development platform. Template build systems are available 
for autoconf projects and for downloading and compiling/packaging third-party 
applications to extend the core root filesystem. 


Like many Raspberry Pi distributions, PiBox comes with a set of scripts to ease 


installation to an SD card. The public releases of the binary distribution includes 
these scripts and all files needed by end users wanting to try PiBox without 
compiling anything. 
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Figure 4. A small but growing set of library APIs was recently added to the software stack. 


Software Stack 


The PiBox Development Platform is a base on which purpose-driven systems can be 
built. For the PiBox Media Server and PiBox Media Player, I’ve developed a UI based on 
Matchbox, a stacked window manager running on X.org over the framebuffer. Apps 
are managed by a demon (appmgr) that handles communications between them 

and a back-end set of daemons (piboxd, for example) for system services. A newly 
developed API layer (Libpibox and Libpnc) provides custom interfaces for handling 
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common tasks, such as network configuration and media sharing. 


The Media systems are created from the Development Platform using a collection of 
opkg-formatted packages. The packages are built from source repositories using a 
cross-compiler environment that relies on the cross toolchain and staging tree from 


the Development Platform. 


PiBox applications are written in C using GTK+ 2.x and Cairo. They install a 
configuration file announcing their presence to the app manager. The app manager 
chooses which apps to show and in what order, then displays them on the navigation 
page scaled to fit the display. Current apps include a themed analog clock, video 
playback, webcam, XMRadio and network configuration, among others. 


The Media systems are designed to be used from the primary console. However, 
additional functionality is available from a web UI, including a webcam stream and 
complete network configuration. The web UI provides user authentication and via 
client-side JavaScript mixed with a PHP back end that speaks a custom network- 
oriented protocol to PiBox demons. This Ul recently received the initial stubs of a 
RESTful interface allowing a custom-designed Android keyboard to be used to control 
the console UI. Android support is still in early development as is a migration to a 
node.js REST API for loT control and home automation. 


All repositories are developed using cdtools, a simple bash configuration tool that 
allows easy context switching between projects while encouraging a common and 
consistent structure for all PiBox-related build systems. 


Video Playback 


The primary purpose of the Media systems is for server-provided video content 
distributed to players. “Players” includes both mobile devices and the custom PiBox 
Media Player. 


Media sources are intended to be found on USB Flash media sticks. The Raspberry 
Pi has its own hardware-accelerated media player, omxplayer. PiBox integrated this 
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American Werewolf In London 
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is nice and quiet small coastal California town yet soon find 
Finding Nemo out that it's overrun by bike gangs and vampires. A couple 


IN of teenage friends take it upon themselves to hunt down 
ce Age the vampires that they suspect of a few mysterious 
Incredibles murders and restore peace and calm to their town. 


Monsters Inc 
Shrek 

Simpsons 

Toy Story 2 

A Christmas Carol 


Figure 5. VideoFE’s video list, which is a sorted collection from multiple sources, is also searchable. 


as a back-end player to a custom front-end Ul component known simply as VideoFE. 
VideoFE is configurable to use other players, and it’s been used on development 
desktops running xine. 


VideoFE presents a list of videos and displays the poster of the current selection. 
Video data comes from The Movie DB via its JASON API. The data is retrieved using a 


custom Java application called VideoLib. The application runs on a desktop. It scans 

a directory tree and does a lookup by filename. The lookup can be customized if the 
filename isn’t a good match, and The Movie DB provides alternative options that also 
can be selected. Once the directory tree is scanned correctly, the database is saved to 
the top of the tree, which should be the top of the media stick. 


The UI has a customizable keymap file that provides a consistent interface for all apps. 
By default, the Esc key exits an application and returns the user to the navigation 
page. The arrow keys move around the navigation page. The design is intended to 
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require as few keys as possible. The actual implementation of the Media systems 
makes use of a very small FAVI keyboard that is ideal for a variety of PiBox use cases. 


Lessons Learned 

As a learning tool, PiBox opened my eyes to a wide variety of problems. The first was 
the problem of switching hardware platforms easily. Crosstool-NG makes this easy, but 
wrapping this inside the PiBox build system made it easy to experiment quickly with 
different configurations. This allowed me to switch originally from the BeagleBone to 


the Raspberry Pi and recently aided in a quick migration to supporting the Raspberry 
Pi 2, which has a slightly different processor configuration. Early versions of PiBox 
toolchains also have been used for PowerPC and Intel targets. 


During early development, | had to deal with device handling during boot. To 
reduce overhead and speed boot times, many embedded systems use mdev instead 
of udev for device handling. Decoding mdev events took some work and led to 
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Figure 6. The console Ul is consumer-oriented, with few keystrokes required for navigation. 
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custom scripting to deal with USB device hot plug events, which were critical to 
the Media systems. 


A related issue invovled removing early boot messages while speeding boot times. 
Boot times are still on the order of a minute or more (mostly due to network 
setup), but boot messages now are handled by the use of psiplash to show boot 


2 IBO Webcam - M: 1a Fire 
File Edit View History Bookmarks Tools Help #@# & OF & + «muse.org v v 


Done $3 6B 


Figure 8. The webcam image can be flipped so the camera can be mounted from above or 
below a window. 
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processing. | also had to start X.org very early without starting any clients. This is due 
to timing issues that prevent keyboards from operating properly. X is later stopped 
and restarted. This circumvents the non-functional keyboard issues without the user 
noticing the workaround. 


PiBox offers both a console Ul-based on GTK+/Cairo and a JavaScript-based interface 
based on the Monkey web server and a PHP back end. | originally looked at building 
an HTMLS interface (and would still like to in the long run), but WebKit wouldn’t 
compile properly, and | didn’t want to waste too much time trying to address that 
issue. So | fell back to what | already knew how to do: build a GTK+-based UI. 


Cairo presented an estoric problem that proved difficult to isolate. Cairo is used by 
GTK+ to paint icons in the navigation page. This code originally was tested on my 
Intel-based desktop and worked fine. On the target board, the icons failed to be 
displayed. After much trial and error, | discovered that Cairo was unhappy within 
non-square images on the ARM board. Making them square fixed the problem. This 
adds an odd requirement for app developers when creating the navigation icons 
for their apps. 


The web UI required me to learn more about JavaScript, a language | enjoy more 
now that | understand it a better. Its interaction with back-end PHP is clean and easy 
to understand, but creating Uls can require carrying client libraries around from 
application to application, much like Java. It does offer the option of setting up 
direct connections from the client browser to non-web-based back ends, and that’s 
something | find very encouraging. 


What | found less encouraging was the use of web-based media through proxies. 
The only format that moves natively through http/https is motion jpeg (mjpeg), 
and on the Pi, this works only at a relatively low frame rate. Note that this is a USB 
webcam, not the onboard camera. | chose to use a webcam due to the relatively 
short ribbon cables provided for the onboard camera. The use case for the 
webcam is as a backup camera for our trailer. The idea is to allow the camera to be 
physically distant from the Pi. 
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Figure 9. pibox-network-config is available in the development platform (above) but takes 
on a different appearance in the Media systems (below). 
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PiBox Media systems are designed to reduce user-managed wires. This includes 
networking, where WiFi rules. Unfortunately, not all WiFi dongles are created 
equal with all versions of the Linux kernel. After many months of trial and error, | 
managed to find that the Ralink RT5370 chip provided the most stable experience. 
However, support was left in place for the variety of chips | tested. Handling USB 
WiFi dongles is done manually, outside of mdev, with the help of USB device IDs 


and a custom init script. 


While researching WiFi dongles, | also improved the network configuration utility | 
wrote from scratch. This GTK+ application provides support for configuring WiFi or 
wired static and DHCP connections and also allows configuring the Media Server as a 
wireless access point for the closed network in our travel trailer. This utility is available 
in the Development Platform, but with GTK+ themes, inherits the look of the custom 


Ul for the Media systems. The utility comes with a WiFi scanner and also provides the 
Libpiboxnet library. The library is used by piboxd, a deemon that handles many 
functions including inbound requests from the web UI. 


Summary 


There are many other issues | covered over the years of PiBox development, including 


learning enough Blender to generate a 3D design for a sample enclosure—an experiment 
still in need of much refinement. But the design of the build system for the Development 


Platform has proven flexible enough for a variety of platforms. Cross-compilation turns 
out to be much simpler than | expected once you get a method identified. | use a 
common cross.sh script in many of the packages | build to simplify this process. | also 
found that the use of cdtools made it possible to create an all-encompassing metabuild 
that automates the download and build process for all packages at release time. 


Future development plans include integration with remote sensors. | hope to create 
my own Iron Man home, with voice commands handled by my Jarvis project. Jarvis is 
a proof-of-concept voice-to-text-to-action-to-speech Java project. But the hope is to 
have it integrate with a back-end sensor management system, connected through a 
PiBox Media Server, to handle control of devices around the home. 
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This entire project has been a one-man job, and I’m very much interested in others 
joining in, even if all they do is fork the project. I’d like to hear how | can improve 
the build systems, the Media systems and their Uls and apps. I’ve created as much 
documentation as | can on the wiki, and interested developers can follow progress 
on my Redmine issue tracker. All code is released as open source and is available 
from GitLab.com. 


Michael J. Hammel is a Software Engineer for NetApp living with his wife Brinda and two Golden Retrievers in Broomfield, Colorado. 
When he isn’t working on embedded systems or other geekery, he likes to camp, walk his dogs around the park, and drink tea with his 
wife and revel in the joy of his daughter’s success. He has written more than 100 articles for numerous online and print magazines, and 
he’s the author of four books on GIMP, the GNU Image Manipulation Program. 
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Text Processing 
in Rust 


Create handy command-line utilities in Rust. 


By Mihalis Tsoukalos 


This article is about text processing in Rust, but it also contains a quick introduction 
to pattern matching, which can be very handy when working with text. 


Strings are a huge subject in Rust, which can be easily realized by the fact that 
Rust has two data types for representing strings as well as support for macros for 
formatting strings. However, all of this also proves how powerful Rust is in string 
and text processing. 


Apart from covering some theoretical topics, this article shows how to develop some 
handy yet easy-to-implement command-line utilities that let you work with plain-text 
files. If you have the time, it’d be great to experiment with the Rust code presented 
here, and maybe develop your own utilities. 


Rust and Text 


Rust supports two data types for working with strings: String and str. The String 
type is for working with mutable strings that belong to you, and it has length and a 
capacity property. On the other hand, the str type is for working with immutable 
strings that you want to pass around. You most likely will see an str variable be used 
as &str. Put simply, an str variable is accessed as a reference to some UTF-8 data. 


An str variable is usually called a “string slice” or, even simpler, a “slice”. Due to its 
nature, you can’t add and remove any data from an existing str variable. Moreover, 
if you try to call the capacity() function onan &str variable, you'll get an error 
message similar to the following: 
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error[E@599]: no method named 'capacity' found for type 
o'&str' in the current scope 


Generally speaking, you'll want to use an str when you want to pass a string as a 
function parameter or when you want to have a read-only version of a string, and then 
use a String variable when you want to have a mutable string that you want to own. 


The good thing is that a function that accepts &str parameters can also accept 
String parameters. (You'll see such an example in the basicOps.rs program 
presented later in this article.) Additionally, Rust supports the char type, which is for 
representing single Unicode characters, as well as string literals, which are strings that 
begin and end with double quotes. 


Finally, Rust supports what is called a byte string. You can define a new byte 
string as follows: 


let a_byte_string = b"Linux Journal"; 


unwrap() 

You almost certainly cannot write a Rust program without using the unwrap () 
function, so let’s take a look at that here. Rust does not have support for null, nil 
or Null, and it uses the Option type for representing a value that may or may not 
exist. If you’re sure that some Option or Result variable that you want to use has a 
value, you can use unwrap() and get that value from the variable. 


However, if that value doesn’t exist, your program will panic. Take a look at the 
following Rust program, which is saved as unwrap.rs: 


use std::net::IpAddr; 
fn main() { 
let my_ip = "127.0.0.1"; 


let parsed_ip: IpAddr = my_ip.parse().unwrap(); 
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printtn!("{}", parsed_ip); 


let invalid ip = "“727.0.0.1"; 
let try_parsed_ip: IpAddr = invalid_ip.parse().unwrap(); 
printin!("{}", try_parsed_ip); 


Two main things are happening here. First, as my_ip is a valid IPv4 address, 
parse().unwrap() will be successful, and parsed_ip will have a valid value 
after the call to unwrap(). 


However, as invalid_ip is not a valid IPv4 address, the second attempt to call 
parse().unwrap() will fail, the program will panic and the second printtn! () 
macro will not be executed. Executing unwrap.rs will verify all these: 


$ ./unwrap 

127 é0eOnt 

thread 'main' panicked at ‘called 'Result::unwrap() ' 
son an ‘Err' 

value: AddrParseError(())', libcore/result.rs:945:5 
note: Run with 'RUST_BACKTRACE=1' for a backtrace. 


This means you should be extra careful when using unwrap() in your Rust programs. 
Unfortunately, going into more depth on unwrap() and how to avoid panic situations 
is beyond the scope of this article. 


The println! and format! Macros 


Rust supports macros, including println! and format! that are related to strings. 


A Rust macro lets you write code that writes other code, which is also known as 
metaprogramming. Although macros look a lot like Rust functions, they have a 
fundamental difference from Rust functions: macros can have a variable number of 
parameters, whereas the signature of a Rust function must declare its parameters and 
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define the exact type of each one of those function parameters. 


As you might already know, the printtn! macro is used for printing output to the 
UNIX standard output, whereas the format! macro, which works in the same way as 
printtn!, returns a new String instead of writing any text to standard output. 


The Rust code of macros.rs will try to clarify things: 


macro_rules! hello_world{ 
() => { 
printtn!("Hello World!") 
yj 


fn double(a: i32) -> i32 { 
return ata 


fn main() { 
// Using the format!() macro 
let my_name = "Mihalis"; 
let salute = format! ("Hello {}!", my_name); 
println!("t}"; salute): 


// Using hello_world 
hello_world!(); 


// Using the assert_eq! macro 


assert_eq!(double(12), 24); 
assert_eq!(double(12), 26); 


What knowledge do you get from macros. rs? First, that macro definitions begin with 
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macro_rules! and can contain other macros in their implementation. Note 

that this is a very naive macro that does nothing really useful. Second, you can 
see that format! can be very handy when you want to create your own strings using 
your own format. Third, the he lLlo_world macro created earlier should be called as 
hello_wortd!(). And finally, this shows that the assert_eq!() macro can help 
you test the correctness of your code. 


Compiling and running macros.rs produces the following output: 


$ ./macros 

Hello Mihalis! 

Hello World! 

thread 'main' panicked at ‘assertion failed: ‘(left == right)' 
left: '24', 
right: °26°" » hacrosarst22'5 

note: Run with 'RUST_BACKTRACE=1' for a backtrace. 


Additionally, you can see an advantage of the assert_eq! macro here: when an 
assert_eq! macro fails, it also prints the line number and the filename of the 
assertion, which can’t be done using a function. 


Working with Strings 
Now let’s look at how to perform basic text operations in Rust. The Rust code for this 
example is saved in basicOp.rs and is the following: 


fn accept(s: &str) { 
print latte, sis 


fn main() { 
// Define a str 
let Lj: &str= "Linux Journal"; 
7/ Ur 
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let magazine: &'static str = "magazine"; 

// Use Tormat! to create a String 

let my_str = format!("Hello {} {}!", lj, magazine); 
printini(™my-str Lit} Cit}", my str. lent), 
omy_str.capacity()); 


// String character by character 

for ¢ in mycstrechars(). 4 
printer ye: cs 

} 

println!t): 


for (i, c) in my_str.chars().enumerate() { 
print Cie yey aa 

} 

printin!()s 


// Convert string to number 

let nt 6str = "19": 

match n.parse::<i32>() { 
Ok(n) => println!("{} is a number!", n), 
Err(e) => printtn!("{} is NOT a number!", e), 


let nist &str = "16.2" 

match nl.parse::<i32>() { 
Ok(n1) => printtn!("{} is a number!", n1), 
Err(e) => println!("4}: 4}", ni; e@), 


// accept() works with both str and String 
Let my_str = “This is str!”: 
let mut my_string = String::from("This is string!"); 
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accept (&my_str); 
accept (&my_string); 


// my_string has capacity 

printtn!("my_string L:{} C:{}", my_string.len(), 
omy_string.capacity()); 

ny string, push str("OK?™): 

printtn! ("my_string L:{} C:{}", my_string.len(), 
omy_string.capacity()); 


// Convert String to str 

let s_str: &str = &my_stringl[..]; 

// Convert str to String 

let s_string: String = s_str.to_owned(); 
printini(™s_ string: Lit} C:{}", S.string.len(), 
os string.capacity()); 


So, first you can see two ways for defining str variables and creating a String 
variable using the format! macro. Then, you can see two techniques for iterating 
over a string character by character. The second technique also returns an index to 
the string that you process. After that, this example shows how to convert a string 
into an integer, if it’s possible, with the help of parse: :<i32>(). Next, you can see 
that the accept() function accepts both an &str and a String parameter even 
though its definition mentions an &str parameter. Following that, this shows the 
capacity and the length properties of a String variable, which are two different 
things. The length of a String is the size of the String, whereas the capacity of a 
String is the room that is currently allocated for that String. Finally, you can see 
how to convert a String to str and vice versa. Other ways for getting a String 
from an &str variable include the use of .to_string(), String::from(), 
String::push_str(), format!() and .into(). 


Executing basicOp.rs generates the following output: 
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$ ./basicOp 

my_str Le29 Cr32 

het Lo Linux Journatl magazine! 
nee erl Liz lis ord GS LG lly mie wis xe1@ 411 3212 ofl 
ou:l4 r:15 

Nt16 ail? TiisS° 219 mi2d aizl. gt22 ‘arzs 2t24 1425 ni26 ef27 
o1:28 

10 is a number! 

10.2: invalid digit found in string 

This is str! 

This is string! 

my_string ILii5 Csi5 

my_string Lii8 C:30 

Ss string! Lite ceie 


Finding Palindrome Strings 

Now, let’s look at a small utility that checks whether a string is a palindrome. 
The string is given as a command-line argument to the program. The logic of 
palindrome. rs is found in the implementation of the check_palindrome() 
function, which is implemented as follows: 


pub fn check_palindrome(input: &str) -> bool f{ 
if input.len() == 0 { 
return true; 


} 
let mut last = input.len() - 1; 
let mut first = Q; 


let my_vec = input.as_bytes().to_owned(); 
while first < last { 
if my_vec[first] != my_vec[last] { 


return false; 
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} 
Tirst +=1: 
last -=1; 


} 


return true; 


The key point here is that you convert the string to a vector using a call to 
as_bytes().to_owned() in order to be able to access it as an array. After that, 
you keep processing the input string from both its left and its right side, one 
character from each side for as long as both characters are the same or until you 
pass the middle of the string. In that case, you are dealing with a palindrome, so 
the function returns “true”; otherwise, the function returns “false”. 


Executing palindrome.rs with various types of input generates the following 
kind of output: 


$ ./palindrome 1 

1 is a palindrome! 

$ ./palindrome 

Usage: ./palindrome string 
$ ./palindrome abccba 
abccba is a palindrome! 

$ ./palindrome abcba 

abcba is a palindrome! 

$ ./palindrome acba 

acba is not a palindrome! 


Pattern Matching 

Pattern matching can be very handy, but you should use it with caution, because it 
can create nasty bugs in your software. Pattern matching in Rust happens with the 
help of the match keyword. A match statement must catch all the possible values of 
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the used variable, so having a default branch at the end of the block is a very common 
practice. The default branch is defined with the help of the underscore character, 
which is a synonym for “catch all”. In some rare situations, such as when you examine 
a condition that can be either true or false, a default branch is not needed. A pattern- 
matching block can look like the following: 


let salute = match a_name 
{ 

"John" => "Hello John!", 
"Jim" => "Hello Boss!", 
VJ” SS "Helle: JTL", 
_ => "Hello stranger!" 


a 


What does that block do? It matches one of the three distinct cases, if there is match, or 
it goes to the match all cases, which is last. If you want to perform more complex tasks 
that require the use of regular expressions, the regex crate might be more appropriate. 


A Version of wc in Rust 

Now let’s look at the implementation of a simplified version of the wc(1) command- 
line utility. The Rust version of the utility will be saved as we.rs, will not support any 
command-line flags, will consider every command-line argument as a file, and it can 
process multiple text files. The Rust version of we.rs is the following: 


use std::env; 
use std::io::{BufReader, BufRead}; 


use std::fs::File; 


fn main() { 


let mut Lines = Q; 
let mut words = Q; 
let mut chars = Q; 


167 | March 2019 | https://www.linuxjournal.com 


TEXT PROCESSING IN RUST 


let args: Vec<_> = env::args().collect(); 

if args.len() == 1 { 
println!("Usage: {} text_file(s)", args[Q@]); 
return; 


let n_args = args.len(); 
for xX in is«targs 4 
let mut total_lines 
let mut total_words = Q; 
let mut total_chars 


II 
S 


II 
S 


let input_path = ::std::env::args().nth(x).unwrap(); 
let file = BufReader::new(File::open(&input_path) 
%,unwrap()); 
for line in file.lines() { 
let my_line = line.unwrap(); 
total_lines = total_lines + 1; 
total_words += my_line.split_whitespace().count(); 
total_chars = total_chars + my_line.len() + 1; 


printin! ("\t{H\t{}\t{F\t{}", total_lines, total_words, 
stotal_chars, input_path); 

lines += total_lines; 

words += total_words; 

chars += total_chars; 


if n_args-1 !=1{ 
printtin! ("\t{#\ti}\t{}\ttotal", lines, words, chars); 
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First, you should know that we. rs is using buffered input for processing its text files. 
Apart from that, the logic of the program is found in the inner for loop that reads 


each input file line by line. For each line it reads, it counts the characters and words. 
Counting the characters of a line is as simple as calling the Len() function. Counting 
the words of a line requires splitting the line using split_whitespace() and 
counting the number of elements in the generated iterator. 


The other thing you should think about is resetting the total_lines, total_words 
and total_chars counters after processing a file. The Lines, words and chars 
variables hold the total number of lines, words and characters read from all 
processed text files. 


Executing wc. rs generates the following kind of output: 


$ rustc wc.rs 

$ ./WC 

Usage: ./wc text_file(s) 
$ ./we wec.rs 


40 124 1114 wc.rs 
$ ./wc wc.rs palindrome.rs 
40 124 1114 wc.rs 
39 104 854 palindrome.rs 
79 228 1968 total 
$ wc wc.rs palindrome.rs 
40 124 1114 wec.rs 
39 104 854 palindrome.rs 
79 228 1968 total 


The last command executed we(1) in order to verify the correctness of the 
output of wc. rs. 


As an exercise, you might try creating a separate function for counting the lines, 
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words and characters of a text file. 


Matching Lines That Contain a Given String 

In this section, you'll see how to show the lines of a text file that match a given 
string—both the filename and the string will be given as command-line arguments to 
the utility, which is named match. rs. Here’s the Rust code for match.rs: 


use std::env; 
use std::io::{BufReader, BufRead}; 
use std::fs::File; 


fn main() { 
let mut total_lines = Q; 
let mut matched_lines = Q; 
let args: Vec<_> = env::args().collect(); 


if args.len() != 3 { 
printtn!("{} filename string", args[Q]); 


return; 
} 
let input_path = ::std::env::args().nth(1).unwrap(); 
let string_to_match = ::std::env::args().nth(2).unwrap(); 


let file = BufReader::new(File: :open(&input_path).unwrap()); 
for line in file.lines() { 
total_lines += 1; 
let my_line = line.unwrap(); 
if my_line.contains(&string_to_match) { 
printtn!("{}", my_line); 
matched_lines += 1; 
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println! ("Lines processed: {}", total_lines); 
println!("Lines matched: {}", matched_lines); 


} 


All the dirty work is done by the contains() function that checks whether the line 
that is currently being processed contains the desired string. Apart from that, the rest 
of the Rust code is pretty trivial. 


Building and executing match. rs generates output like this: 


$ ./match tabSpace.rs t2s 

Th t2s(input? Gstr, me 132) 4 
t2s(&input_path, n_space); 

Lines processed: 56 

Lines matched: 2 

$ ./match tabSpace.rs doesNotExist 

Lines processed: 56 

Lines matched: @ 


Converting between Tabs and Spaces 
Next, let’s develop a command-line utility that can convert tabs to spaces in a text file 
and vice versa. Each tab is replaced with four space characters and vice versa. 


This utility requires at least two command-line parameters: the first one should 
indicate whether you want to replace tabs with spaces or the other way around. After 
that, you should give the path of at least one text file. The utility will process as many 
text files as you want, just like the wc. rs utility presented earlier in this article. 


You can find tabSpace.rs’s logic in the following two Rust functions: 
fi t2s(input: &str) 4 
let file = BufReader: :new(File: :open(&input).unwrap()); 


for line in file. lines() 4 
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let my_line = lLine.unwrap(); 
let new_line = my_line.replace("\t", " my 
printtn! ("{}", new_line); 


fn s2t(input: &str) { 
let file = BufReader::new(File: :open(&input).unwrap()); 
for line in file.lines() { 
let my_line = line.unwrap(); 
let new_line = my_line. replace(" Py eee 
printtn! ("{}", new_line); 


All the work is done by replace(), which replaces every occurrence of the first 
pattern with the second one. The return value of the replace() function is the 
altered version of the input string, which is what’s printed on your screen. 


Executing tabSpace.rs creates output like the following: 


$ ./tabSpace -t basicOp.rs > spaces.rs 
Processing basicOp.rs 

$ mv spaces.rs basicOp.rs 

$ ./tabSpace -s basicOp.rs > tabs.rs 
Processing basicOp.rs 

$ ./tabSpace -t tabs.rs > sSpaces.rs 
Processing tabs.rs 

$ diff spaces.rs basicOp.rs 


The previous command verifies the correctness of tabSpace. rs. First, any tabs in 
basicOp.rs are converted into spaces and saved as spaces.rs, which afterward 


becomes the new basicOps.rs. Then, the spaces of basicOps.rs are converted into 


172 | March 2019 | https://www.linuxjournal.com 


TEXT PROCESSING IN RUST 


tabs and saved in tabs.rs. Finally, the tabs.rs file is processed, and all of its tabs are 
converted into spaces (spaces.rs). The last version of spaces.rs should be exactly the 
same as basicOps.rs. 


It would be a very interesting exercise to add support for tabs of variable size in 
tabSpace.rs. Put simply, the number of spaces of a tab should be a variable that will 
be given as acommand-line parameter to the utility. 


Conclusion 

So, is Rust good at text processing and working with text in general? Yes it is! 
Additionally, it should be clear that text processing is closely related to file I/O and 
(sometimes) to pattern matching and regular expressions. 


The only rational way to learn more about text processing in Rust is to experiment on 
your own, so don’t waste any more time, and give it a whirl. m 


Mihalis Tsoukalos is a UNIX administrator and developer, a DBA and mathematician who enjoys technical writing. He is the author of 
Go Systems Programming and Mastering Go. You can reach him at http://www.mtsoukalos.eu and @mactsouk. 
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other side of things by writing good open-source code that 
supports and enhances open science. 


People working in science potentially can benefit from every 
piece of free software code—the operating systems and apps, 
and the tools and libraries—so the better those become, the 
more useful they are for scientists. But there’s one open-source 
project in particular that already has had a significant impact on 
how scientists work—Project Jupyter: 


Project Jupyter is a set of open-source software projects 
that form the building blocks for interactive and exploratory 
computing that is reproducible and multi-language. The main 
application offered by Jupyter is the Jupyter Notebook, a 
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web-based interactive computing platform that allows users to author documents that 
combine live code, equations, narrative text, interactive dashboard and other rich media. 


Project Jupyter was spun-off from IPython in 2014 by Fernando Pérez. Although 

it began as an environment for programming Python, its ambitions have grown 
considerably. Today, dozens of Jupyter kernels exist that allow other languages to be 
used. Indeed, the project itself speaks of supporting “interactive data science and 
scientific computing across all programming languages”. As well as this broad-based 
support for programming languages, Jupyter is noteworthy for its power. It enables 
users to create and share documents that contain live code, equations, visualizations 
and narrative text. Uses include data cleaning and transformation, numerical 
simulation, statistical modeling, data visualization and machine learning. 


In a way, Project Jupyter is the ultimate scientific tool, since it can be used in any 
discipline and for multiple purposes. As an article in the Atlantic rightly put it, it also 
can be thought of as the scientific paper taken to the next level by exploiting the 
possibilities of digital technology. A key aspect is that it’s interactive—readers can use 
the embedded code to explore the data and carry out limitless “what ifs”. It’s such an 
obvious idea, you may wonder why it hasn’t been done before. And the answer is that 
it has, notably in the form of Mathematica from Wolfram Research. 


Mathematica is an innovative and powerful program with one huge flaw: it’s 
proprietary. As such, it suffers all the usual downsides, one of which more or less 
disqualifies it for science: you can’t check the code. That means you don’t really 
know why it produces the results it does; you just have to take it on trust. That’s not 
science; that’s voodoo. 


Its closed-source nature means that Mathematica can’t tap into the community of 
users in the same way open-source projects can. Whatever advantages Mathematica 
once had, it’s only a matter of time before open-source alternatives like Jupyter 
surpass it. Indeed, it’s interesting that the Google Trends comparison of searches 

for Mathematica and searches for Jupyter show that interest in the latter is rising, 
while Google searches for the former are falling. It’s an inexact metric, of course, but 
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the overall trends are clear: Mathematica, like Microsoft Windows, is the past, and 
Jupyter, like GNU/Linux, is the future. 


It’s not just about the familiar dynamics of open-source development. There’s a key 
reason why Jupyter has beaten Mathematica, as the academic Paul Romer explained 
in a perceptive post: 


Mathematica failed, despite technical accomplishments, because the norms of its 
developers clashed so obviously with the norms of its intended users. Jupyter is 
succeeding because the norms of the community that is developing it are aligned with 
the norms of its users. 


As well as its culture, there’s another aspect of Jupyter that makes it a perfect 
fit for open science. On the page listing dozens of Jupyter notebooks—all freely 
accessible—there’s a section titled “Reproducible academic publications”: 


This section contains academic papers that have been published in the peer-reviewed 
literature or pre-print sites such as the ArXiv that include one or more notebooks that 
enable (even if only partially) readers to reproduce the results of the publication. 


Coupled with the transparency of the underlying code, this ability for anybody to 
check the logic and results of a publication is a real breakthrough in open science. At 
the moment, most academic papers can be read only superficially. In theory, anyone 
could set about reproducing the final conclusions—at least, provided the relevant 


datasets are freely available. Few will take the trouble to do so though, because there 
are no academic incentives to expend all that time and energy. With papers published 
not as static documents, but as dynamic Jupyter notebooks with full open datasets, it 
is possible to check the results properly, as well as to plug in other datasets or tweak 
the underlying assumptions. In this way, Jupyter notebooks are the perfect marriage 
of open source, open access and open data. This is exactly how open science should 
work, but until now almost never does. 


The power and flexibility of the Jupyter environment make it a strong foundation 
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for open-ended experimentation of the kind the Free Software community relishes. 
Moreover, coding in this domain could have a major impact on scientists using the 
notebook format and on the science they produce. That combination of satisfying 
intellectual challenge with real-world practical benefits makes it a perfect candidate 
for open-source coders looking for new and meaningful challenges. 
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